About 10 days ago I showed you how to build a simple Yahoo Pipe to scrape the Google Hot Trends. Today, I am going to take that tutorial to the next level.
You might want to use the hot keywords in your CPV campaign. Rather than manually copying each of the top 10 results, you should really automate this task.
Unfortunately, Yahoo Pipes does not work with Google.
The reason is that Yahoo Pipes supports the robot.txt setting which asks ‘spiders’ not to scrape.
No worries though. There is another excellent tool that you can use:
So today I will show you how you can easily scrape URLs from Google with Dapper. Its a simple and quick method that still works, even after Google switched on Instant Search.
Note: throughout the next steps described below, a window can pop up offering you a demo of the next step in the process. Feel free to watch these to get to know how dapper works.
1. Switch off Google Instant
To scrape URLs with Dapper you will need a google search URL. You dont want to use the one from Google Instant. So switch it off by starting to type something in the search box. Then select the option to switch it off:
Next, click the Advanced Search button, and select the option to return 100 results. Do your random search and grab the Google URL once the results are showing. If you want you can clean up the URL, but its not essential.
Note down the URL in a text document as you will need it later.
2. Create the Dapp
Next, log in to Dapper, and create a new Dapp. Then just paste the Google URL from the previous step into the ‘Enter the URL’ field and press ‘Next step’.
This will then show you a window with the google search screen, where in the next step you will save a couple of pages.
3. Saving pages
To ensure that Dapper scrapes the right information, you need to teach it what text to grab. You will want to grab a couple of pages to make sure it does this correctly.
You need to add the page to dapper by clicking the ‘Add to Basket’ button. Then enter a different result in the search box, and press seach. Again, add this page to the basked. Do this once more.
Then click ‘Next Step’ in the left side bar.
4. Selecting content
You are now at the 3rd stage of the dapper scraper, where you select the content you want to scrape.
All you need to do is click the title of the first search result to select the content you want to scrape:
5. Preview feed (nothing to do here)
6. Saving & configuring your Dapp
Then, under the input variables heading, click edit next to ‘Query’. This will open a input box with the google URL. You will need to edit this URL to identify the query variable. The full url is below and the bits you want to edit:
q=searchkeyword
num=100
In the URL, replace this with {url} and {num}.
Click save next to the input box, and this should show the URL, with {url} and {num} highlighted in green, which means this is treated as an input variable.
Then click the Save button in the bottom right of the screen, and you will be taken to the Dapp results screen.
7. Getting ready to use your Dapp
Next, you will need to choose your format. It is set to XML as default, so all that you need to do is click go to go to the screen to configure the xml feed.
What this does is create a URL that will load the search results and return them as a simple XML feed. This way you can easily import the results in Yahoo Pipes. Make sure you add some easily recognisable text to the url field, and click ‘Update Input’. This will update the XML url and add the required variables.
