A User-Agent
is a bunch of text that is sent with every HTTP and HTTPS request. The server processing this request is able to determine what type of device and browser has made the request.
Often times servers use this parameter to restrict access to the resource.
However, it’s easy to fake a User-Agent when using Python to make HTTP and HTTPS requests.
Using Requests Library
import requests user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36' response = requests.get('https://ao.gl', headers={'User-Agent': user_agent}) html = response.content print(response.content)
Using URLLib Library
import urllib.request user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36' request = urllib.request.Request('https://ao.gl', headers={'User-Agent': user_agent}) response = urllib.request.urlopen(request) html = response.read()