A User-Agent is a bunch of text that is sent with every HTTP and HTTPS request. The server processing this request is able to determine what type of device and browser has made the request.
Often times servers use this parameter to restrict access to the resource.
However, it’s easy to fake a User-Agent when using Python to make HTTP and HTTPS requests.
Using Requests Library
import requests
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
response = requests.get('https://ao.gl', headers={'User-Agent': user_agent})
html = response.content
print(response.content)
Code language: Python (python)Using URLLib Library
import urllib.request
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36'
request = urllib.request.Request('https://ao.gl', headers={'User-Agent': user_agent})
response = urllib.request.urlopen(request)
html = response.read()
Code language: Python (python)