Use different search crawler robot agents as well as LLVM to simulate visiting sites.