Crablo
4 lines
63 B
Plaintext
4 lines
63 B
Plaintext
# allow crawling everything by default
|
|
User-agent: *
|
|
Disallow:
|
# allow crawling everything by default
|
|
User-agent: *
|
|
Disallow:
|