What a 304 response remembers
An unchanged response can save a download. It does not preserve every reason an earlier answer was useful.
The readable web
Reading the web’s crawl rules
robots.txt, search indexing, and authentication govern different decisions. Three primary documents explain the boundaries.
Open the sourcebook →Fetch a pagerobots.txt
List it in searchindexing directives
Read private contentauthentication
Each answers a different question.