This is a response to a query regarding Eric Archambault's report on OA Growth by Adam G Dunn in Science Insider: "I find it difficult to believe that the authors of the study managed to create a harvester that could identify and verify the pdfs linked to by Google Scholar when Google Scholar actively blocks IP addresses when they identify crawling."

Our own "harvester" attempts to gather the all-important data on OA growth were blocked by Google. 

It is completely understandable and justifiable that Google shields its increasingly vital global database and search mechanisms from the countless and incessant worldwide attempts at exploitation by commercial interests, spammers, and malware that could bring Google to its knees if not rigorously and relentlessly blocked. 

But in the very special (and tiny) case of scientific research articles it would not only be a great help to the worldwide research community but to Google (and Google Scholar) itself if Google granted special individual exemptions for important international studies like Eric Archambault's, which was commissioned by the European Union to monitor the global growth rate of open access to research. 

Google and Google Scholar would become all the richer as research databases if data like Eric's (and our own) were not made so excruciatingly difficult and time-consuming to gather by Google's blanket blockage of automated data-mining.

(We do not trawl books, so Google's agreements with publishers are not violated or at issue in any way. We just want to trawl for articles whose metadata match the the metadata from Web of Science or SCOPUS and have been made freely accessible on the web; nor do we want their full-texts: just to check whether they are there!)

Stevan Harnad