The website with a lot of rich content that you would like to crawl may be behind a login authentication page.
To bypass this login authentication, you have to follow the steps below and also download an extension. Please also ensure that you have the full authority and right to crawl a password protected site.
Step 1: Download a Chrome Extension
Download this chrome extension which would help you to fetch a session cookie allowing Wonderchat access to crawl your site.
Get cookies.txt LOCALLY
- The tool downloads cookies locally into your server so it would allow you to safely store the cookie.
- Click on “Add to chrome”
- The extension should now show up on your side bar
Step 2: Log into your private website
- Go to your website, and log into the password protected site.
- For example, we want to crawl a Wordpress community site so we have to be logged into the website.
- Ensure that you are logged into the site
Step 3: Use the Cookies extension within your logged in private website