Webflow Experts Scraper
Finding leads can be difficult, but online directories can be a great first step in building connections. In this example template, we show you how to use Autocode's built in crawler API combined with the power of regex to automatically scrape potential leads from the Webflow Experts page and store them in a Google Sheet. You can modify this code and apply the concepts within it to most online directories to collect the data thats most value to you.
How do I use this template?
- Create a blank Google Sheet
- On the Google Sheet, create a tab called
Experts
- On the
Experts tab, add the following headings to the first row:
name, profileUrl, description, tags, expertSince, location, languages, website
- Install this template using the green
Install Free button at the top of this page
- Link the Google Sheet you just created to your project when prompted
- Once the project has finished instaling, open the
populate-sheet.js file
- Click the
Tools button in the bottom right of the editor. Then open the Payload tab.
- Set the
startPage and endPage for the scraper. For example, to start scraping at page 13 and finish scraping at page 20, you'd set it to {"startPage": 13, "endPage": 20} At the time of writing there were 20 pages. I recommend you only scrape around 20 pages at at time to avoid errors caused by toggling/anti-DDoS protection.
- Click the green
Run button in the bottom right corner of the editor.
- Wait for the scraper to finish running and check your sheet! You should see a bunch of Webflow Experts to reach out to as leads. Then, repeat for other pages as neccesary.
How does this template work?
Once you run the /functions/populate-sheet.js endpoint, the script will automatically loop through the pages you set in the payload. For each page, it will scrape the experts on that page by finding HTML elements with the class .experts-item. Then, for each expert found, it will scrape that expert's Webflow profile page to save addition information, including the expert's website and location. Once all of this data has been scraped it is automatically added to a Google Sheet.
What does each file do?
/functions/populate-sheet.js - run this file! This is the file which calls the other endpoints and then saves the returned data to a Google Sheet./functions/scape-experts.js - this file scrapes the list of experts contained within a single Webflow Experts directory page then returns them as JSON. This file is called by /functions/populate-sheet.js./functions/get-profile-data - this file scrapes data from an individual experts profile page such as website, location, and how long they've been an expert for. This file is called by /functions/populate-sheet.js.
FAQ
- My script times out when I run
/functions/populate-sheet.js! How come?
- You're probably trying to scrape a lot of experts at once. Try scraping less pages in one request, or alternatively upgrade to a paid Autocode plan then set a higher timeout in the bottom left corner of the editor.
- I'm getting an error whenever I try to scrape data? What's the likely cause of this?
- If you try to scrape a lot of data at once, this can occasionally cause an error. Try scraping less pages at once.
- Can I customize this code to scrape another website?
- Yes! Feel free to customize this code to scrape another service however you wish. Alternatively, you can try building your own project from scratch using Autocode's crawler API.
