Table of Contents

As you might already know the Official Google Places API is limited to 5 reviews per one place only. Therefore developers are looking into scraping to have the ability to fetch all the reviews from any business on Google Maps.

Scraping Google with all its protections and dynamically rendering pages might be a challenging task. Fortunately, there are many tools that you can use to scraper reviews in python or any other programming language. In this blog post, you will see the two most common tools of scraping Google Reviews: browser emulation and Outscraper Platform. Each of them is sufficient to get all the reviews from any listing from maps.

Scraping Google Reviews in Python by Using Browser to Render Dynamic Content

We will use Selenium to control the Chrome browser. The browser will render the dynamic pages of Google Reviews. To get started with building the reviews scraper with Selenium we'll need:

  1. Python 3+.
  2. Chrome browser installed.
  3. Selenium 3.141.0+ (python package).
  4. Chrome Driver (for your OS).
  5. Parsel or any other library to extract data from HTML like Beautiful Soup.

Install Selenium And Other Packages

Install Selenium and Parsel packages by running the following commands. We will use Parsel later when we will parse content from HTML.

				
					pip install selenium
pip install parsel # to extract data from HTML using XPath or CSS selectors
				
			

Start The Browser

Before atarting the driver make sure you did the previous steps and you have the path to your chromedriver file. Initialize the driver by the following code. You should see the new browser window opens.

				
					from selenium import webdriver


chromedrive_path = './chromedriver' # use the path to the driver you downloaded from previous steps
driver = webdriver.Chrome(chromedrive_path)
				
			

On mac you might see the following: "chromedriver cannot be opened because the developer cannot be verified". To overcome this control-click the chromedriver in Finder, choose Open from the menu, and then click Open in the dialog that appears. You should see "ChromeDriver was started successfully" in the opened terminal windows. Close it and after this, you will be able to start ChromeDriver from your code.

Download All Reviews Page

Once you start the driver you are ready to open some pages. To open any page use the "get" command.

				
					url = 'https://www.google.com/maps/place/Central+Park+Zoo/@40.7712318,-73.9674707,15z/data=!3m1!5s0x89c259a1e735d943:0xb63f84c661f84258!4m16!1m8!3m7!1s0x89c258faf553cfad:0x8e9cfc7444d8f876!2sTrump+Tower!8m2!3d40.7624284!4d-73.973794!9m1!1b1!3m6!1s0x89c258f1fcd66869:0x65d72e84d91a3f14!8m2!3d40.767778!4d-73.9718335!9m1!1b1?hl=en&hl=en'
driver.get(url)
				
			

Parse Reviews

Once your page is opened you will see the page in your Chrome window that is controlled by your code. You can run the following code to get the HTML page content from the driver.

				
					page_content = driver.page_source
				
			

To comfortably see the HTML content open the developer console in Chrome by opening the Chrome Menu in the upper-right-hand corner of the browser window and selecting More Tools > Developer Tools. Now you should be able to see elements of your page.

scrape google reviews in python
Finding xpath to the reviews we want to get with developer console

You can parse the content from the HTML page by using your favorite parsing tools. We'll use Parsel in this tutorial.

				
					from parsel import Selector

response = Selector(page_content)
				
			

Iterate over the reviews.

				
					results = []

for el in response.xpath('//div/div[@data-review-id]/div[contains(@class, "content")]'):
    results.append({
        'title': el.xpath('.//div[contains(@class, "title")]/span/text()').extract_first(''),
        'rating': el.xpath('.//span[contains(@aria-label, "stars")]/@aria-label').extract_first('').replace('stars' ,'').strip(),
        'body': el.xpath('.//span[contains(@class, "text")]/text()').extract_first(''),
    })
    
print(results)
				
			

Output from the Google Reviews Crwaler (shortened).

				
					[
  {
    'title': 'Wanda Garrett',
    'rating': '5',
    'body': 'Beautiful ✨ park with a family-friendly atmosphere! I had a great time here; seeing all of the animals and learning all of the interesting facts was a fantastic way to spend the day. The zoo is beautifully landscaped and surrounded by …'
  },
  {
    'title': 'Bernadette Bennett',
    'rating': '4',
    'body': 'Worth going for the seals! They are the main attraction and located in the center of the zoo. We watched a live feeding and it was great. The kids loved it. The zoo is well manicured surrounded by gorgeous gardens. Lots of benches to rest …'
  },
  {
    'title': 'Mary Cutrufelli',
    'rating': '3',
    'body': "So not gonna lie... We came from PA. My kid expected to see lions and hippos and zebra from Madagascar. None of that which is there. It's clean it's a nice zoo. I wouldn't go again though."
  },
  ...
]
				
			

Stop The Browser

It's important to start and stop the driver before and after the scraping accordingly. It's the same as you would open and close your browser before and after surfing the internet. Stop the browser by running the following code.

				
					driver.quit()
				
			

Despite the tricky HTML structure of Google Reviews, with Selenium and good knowledge in XPath and CSS selectors you can achieve quite good results in scraping. This method of using a browser emulator should protect you from getting blocked. However, if you are going to scale your application you might consider using proxies as well in order to avoid some unexpected problems.

Multiprocessing And Other Recommendations

It's possible to run drivers in multiprocessing (not multithreading) but each driver will consume one CPU. Make sure you have enough of them.

The Easiest Way of Scraping Google Reviews in Python

Extracting data from Google with browsers has its pros and cons. Although, you can develop the scraper by yourself, during scaling, it might lead to big expenses for using servers with huge amounts of CPUs in order to handle browsers emulations. Additionally, there should be a person who maintains the crawler and updates it during Google site changes.

By using Outscraper Platform, API, or SDKs Outscraper provides the easiest solution for businesses and individuals to start Scraping Reviews from Google without handling proxies, browser emulation, and investing into developing.


Scrape Reviews In Python By Using SDK

1. You will need python3+ and this python package. Install the package by running the command.

				
					pip install google-services-api
				
			

2. Get your API key from the Profile page.

3. Import the package and initialize it with the key.

4. Specify the location by providing a link, place Id, or name.

				
					from outscraper import ApiClient


api_cliet = ApiClient(api_key='KEY_FROM_OUTSCRAPER')
response = api_cliet.google_maps_reviews(
    'https://www.google.com/maps/place/Do+or+Dive+Bar/@40.6867831,-73.9570104,17z/data=!3m2!4b1!5s0x89c25b96a0b10eb9:0xfe4f81ff249e280d!4m5!3m4!1s0x89c25b96a0b30001:0x643d0464b3138078!8m2!3d40.6867791!4d-73.9548217',
    language='en',
    limit=100
)
				
			

5. Wait a few seconds till the reviews will be fetched.

				
					[
    {
        "name": "Do or Dive Bar",
        "full_address": "1108 Bedford Ave, Brooklyn, NY 11216, United States",
        "borough": "Bedford-Stuyvesant",
        "street": "1108 Bedford Ave",
        "city": "Brooklyn",
        "postal_code": "11216",
        "country_code": "US",
        "country": "United States of America",
        "us_state": "New York",
        "state": "New York",
        "plus_code": null,
        "latitude": 40.686779099999995,
        "longitude": -73.9548217,
        "time_zone": "America/New_York",
        "site": "https://www.doordivebedstuy.com/",
        "phone": "+1 917-867-5309",
        "type": "Bar",
        "rating": 4.5,
        "reviews": 425,
        "reviews_data": [
            {
                "google_id": "0x89c25b96a0b30001:0x643d0464b3138078",
                "autor_link": "https://www.google.com/maps/contrib/115539085325450648866?hl=en-US",
                "autor_name": "Sam Grjaznovs",
                "autor_id": "115539085325450648866",
                "autor_image": "https://lh3.googleusercontent.com/a-/AOh14GgxmEH7a10v6Bo8AFb6OkbyxxfIBPXbMYVAxeSIRA=c0x00000000-cc-rp-ba3",
                "review_text": "Cozy shin dig with an assortment of drinks. They have a strong specialty for 10bucks and merch too. They have out side dining as well as back yard area. Ask for Brandon every other Saturday. He\u2019s hella cute!",
                "review_img_url": "https://lh5.googleusercontent.com/p/AF1QipPNs8QvvdkBonV5wuxdoylFjLY3k7L6muepbDq-",
                "owner_answer": null,
                "owner_answer_timestamp": null,
                "owner_answer_timestamp_datetime_utc": null,
                "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s115539085325450648866!2s0x0:0x643d0464b3138078?hl=en-US",
                "review_rating": 5,
                "review_timestamp": 1603781021,
                "review_datetime_utc": "10/27/2020 06:43:41",
                "review_likes": 0,
                "reviews_id": "7222934207919784056"
            },
            {
                "google_id": "0x89c25b96a0b30001:0x643d0464b3138078",
                "autor_link": "https://www.google.com/maps/contrib/110571545135018844510?hl=en-US",
                "autor_name": "Arabella Stephens",
                "autor_id": "110571545135018844510",
                "autor_image": "https://lh3.googleusercontent.com/a-/AOh14GisqDfheDO0Aq0cu1Z7YBTbzLyvSEvM3IMDKg3q=c0x00000000-cc-rp",
                "review_text": "Great atmosphere, always fun vibe and good beers. I live in the area and this is a very reliable standby. Would recommend to anyone who is not pretentious and likes a bit of clutter in their watering hole.",
                "review_img_url": "https://lh3.googleusercontent.com/a-/AOh14GisqDfheDO0Aq0cu1Z7YBTbzLyvSEvM3IMDKg3q",
                "owner_answer": null,
                "owner_answer_timestamp": null,
                "owner_answer_timestamp_datetime_utc": null,
                "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s110571545135018844510!2s0x0:0x643d0464b3138078?hl=en-US",
                "review_rating": 5,
                "review_timestamp": 1614111762,
                "review_datetime_utc": "02/23/2021 20:22:42",
                "review_likes": 0,
                "reviews_id": "7222934207919784056"
            },
            {
                "google_id": "0x89c25b96a0b30001:0x643d0464b3138078",
                "autor_link": "https://www.google.com/maps/contrib/101725757133547547783?hl=en-US",
                "autor_name": "Jack Parker",
                "autor_id": "101725757133547547783",
                "autor_image": "https://lh3.googleusercontent.com/a-/AOh14GjFK9CLb8__u5PtJzH1rGuX4DVgPvjaEeIkSJnCNw=c0x00000000-cc-rp",
                "review_text": "All the bartenders are rad. Cheap drinks, and a nice backyard. They have space heaters, but I would still recommend bundling up if you plan on spending a while there. Jeopardy night is always fun too. Can\u2019t wait to sit inside again!",
                "review_img_url": "https://lh3.googleusercontent.com/a-/AOh14GjFK9CLb8__u5PtJzH1rGuX4DVgPvjaEeIkSJnCNw",
                "owner_answer": null,
                "owner_answer_timestamp": null,
                "owner_answer_timestamp_datetime_utc": null,
                "review_link": "https://www.google.com/maps/reviews/data=!4m5!14m4!1m3!1m2!1s101725757133547547783!2s0x0:0x643d0464b3138078?hl=en-US",
                "review_rating": 5,
                "review_timestamp": 1611947492,
                "review_datetime_utc": "01/29/2021 19:11:32",
                "review_likes": 0,
                "reviews_id": "7222934207919784056"
            },
            ...
        ]
]
				
			

Video Tutorial on How to Scrape Google Maps Reviews in Python


Extra

Sign up now and use the promotion code OUTSCRAPER1 to get a 10% OFF discount.

Python package ► https://pypi.org/project/google-services-api
API ► https://app.outscraper.com/api-docs

Vlad

Project Manager Linkedin

en_US