Burf.com

After months of work, new ideas and starting again we believe we have the fixed stability issues with Burf.com. No more will you do a search, wait 10 seconds and be returned an error. The answer was down to clustering of the search index into 37 different tables to spread the load. This has resulted in a small decrease in speed in indexing sites and producing search results however will allow for a greater uptime and a much larger index!

The search engine has a common problem, which is Quality vs. Speed. The more relevant the search results get, the longer they take to produce. One solution, which I was going to go with, was to throw more hardware at it! Though this would work, it is avoiding the issue, which could be summed up as poor design.

So do I go with the quicker approach of adding more hardware OR do I rewrite the engine from scratch to meet my needs (engine was first designed to handle only 1 million sites). Well what I plan to do is a little of both. Over the next couple of weeks I will be tweaking the engine to produce better quality results for cached searches and tweaking the non-cached results to return at a fast rate. Adding extra hardware to make sure I have redundancy and better caching should prove beneficial too.

The only pothole I will fall into is that a non-cached result could be rubbish and scare off the user. This will be something I will have to monitor. So far the lack of speed has managed to scare many people off!

If you have not noticed, Burf.com is now using the SearchFeed.com advertising feed within its search engine results page. It is directly embedded in to the main page and not the right hand column where you would typically see Adverts. I have made the feed look like it is part of the results and not sponsored links.

Why you might ask?
Though this does products some revenue, I am more interested in how people use it. So far it looks like 1% of people click on Adverts when they release that it is a paid adverts. This is only marginally higher (0.8) than when they were marked as Sponsored Adverts! (Last weeks test) I am not sure why this is because I would expect the click thru rate to be a lot higher.

The next test is to put the feed in to the right column and see what results that produces. Will people think its Google Adsense and assume it is extremely relevant and then click more? I expect there to be a huge drop in click thru’s

We are currently working on a search webservice and would like any idea’s on what functionality users would want like filters and functions. We are looking to release a basic search results service by the end of the week. This will allow people to query and retrieve data from Burf.com in a xml based format.

To help improve the quality of our search results and to give the visitors more control, you can now remove a site from our listing via “Ban Site” link listed next to each site on the search results page. If you believe the site is not a suitable one, then please ban it. All Banned sites will instantly be removed from the listing until we can verify that it should or should not be removed

We hope this brings a better quality of results to our users

The new search engine (beta) currently uses the MS Indexing Server (the previous used a big database I wrote but did not work too well). The indexing service acts as a webservice (.net) for my main program (url recorder).

The main program has a database or URL which it recieves in 2 ways, submitted and crawled via other pages. These urls are crawled (to find more urls) and if the page is usable then it is stored in the indexing service (i know i probably hit file system problems soon).

My spider is written in .net and basically is a small multithreaded program that takes a log of X number of urls and returns the page data which then the main program extracts the urls.

Because its a new engine, it on only has about 100,000 pages indexed but I hope to hit 1 million by the end of the week.

It currently sites on (due for a change) on a AMD 64 3000 with 1gb of ram and 160gb Sata raid drives (striped)

Last night I recieved over 60,000 urls submitted and not many searches (Sad) If this new engine seems popular then I hope to get a faster connection and machine soon.

Well most search engines would not tell you this information but we have decided to. As this project progresses this can change but here is how we basically rate your page. We use old style page ranking.

We index your title, url, and meta description / keywords. If a word is found in all 4 of these places (like news) then we will give your page a page rank of 4. We also have a program that looks at the whole page and then returns 10 keywords which will give you a top rating of 5 for any word

It is a very simple way to rate a page but seems to be giving good results. We will soon be adding a url length process to help give actural ratings.

06/05/06 Simon