Hardware vs. efficient design
The search engine has a common problem, which is Quality vs. Speed. The more relevant the search results get, the longer they take to produce. One solution, which I was going to go with, was to throw more hardware at it! Though this would work, it is avoiding the issue, which could be summed up as poor design.
So do I go with the quicker approach of adding more hardware OR do I rewrite the engine from scratch to meet my needs (engine was first designed to handle only 1 million sites). Well what I plan to do is a little of both. Over the next couple of weeks I will be tweaking the engine to produce better quality results for cached searches and tweaking the non-cached results to return at a fast rate. Adding extra hardware to make sure I have redundancy and better caching should prove beneficial too.
The only pothole I will fall into is that a non-cached result could be rubbish and scare off the user. This will be something I will have to monitor. So far the lack of speed has managed to scare many people off!
October 28th, 2006 at 8:21 am
There is a tradeoff between quality and speed. Why not cooperate with Trafficproducer at WPW? He also knows assembler. He has a lot of quality links.
There is much about datastructures and efficient algorithmes. I know red / black trees and skip lists are good datastructures. Can these be combined with effective search in linear timle like radix or bucket sort? There is a relation between sorting and searching. You should use paging results to produce the SERP’s that is much faster now, but still of poor quiality, esepcially in the profession I know best, finance. What about distributed objects and hosted applications?
Kjell Bleivik