• Skip to primary navigation
  • Skip to main content
  • Skip to footer

Oh Well

  • About This Blog

Robots

26 August 2004 by Tim

There has been a discussion elsewhere on the ring about search engines spidering blogs.

For instructions on how to block search engine bots This Page covers reasonably clearly how to do it.

But if you simply want to block all bots all you need to do is create a text file containing the following two lines:

User-agent: *

Disallow: /

Save it with the name robots.txt

Upload it to the root of your site so that it sits alongside your home page.

Filed Under: General

« Online School
Looking down on people »

Reader Interactions

Comments

  1. Merry says

    26 August 2004 at 18:19

    Gosh, you’re good. :~)

  2. jax says

    26 August 2004 at 18:37

    Harumph, using my blog to show off! Doesn’t he know that’s my job here? ๐Ÿ˜‰

  3. Barbara says

    26 August 2004 at 19:01

    Totally didn’t understand your comment, Jax, because I assumed you had posted this post. I would have just kept quite ๐Ÿ˜‰

  4. Merry says

    26 August 2004 at 19:15

    kept quite what? ;~)

  5. Barbara says

    26 August 2004 at 19:21

    :-P~~~~~~~~~~~~~~~~~~~

  6. Jonathan says

    26 August 2004 at 20:05

    What about just blocking the blog section of Barbara’s site. An eg for that would be good too ๐Ÿ™‚

  7. Tim says

    26 August 2004 at 20:10

    I supplied the url, so if people want more complicated rules, they can find out how to do them easily enoug, by reading the examples there. And they can’t blame me if they end up blocking off all their traffic. ๐Ÿ™‚

  8. Merry says

    26 August 2004 at 20:40

    rofl – you always say those sites are straight forward and when i go look they always appear to be in ancient greek!

  9. Sarah, Roo and Kids says

    26 August 2004 at 21:12

    surely that should be ancient GEEK?

  10. Tim says

    27 August 2004 at 00:21

    B***dy charming. Who are you calling ancient!!!

  11. Merry says

    28 August 2004 at 10:21

    giggles away loopily!

Footer

Archives

Categories

Search

Copyright © 2025 ยท Tim Marchant ยท Cookie Policy ยท Privacy Policy ยท Log in