News Focus
News Focus
Followers 210
Posts 7903
Boards Moderated 15
Alias Born 05/24/2001

Re: occams_razor post# 52005

Thursday, 03/17/2005 12:16:41 PM

Thursday, March 17, 2005 12:16:41 PM

Post# of 222342
It won't get reset every January 1st. I plan to do the reset in March of every year. Preferably the end of the month so that no search table ends up being less than 3 months.

Making a sliding one-year window would be possible and I was doing this on SI for a while, but it turned out to be too expensive and too easily broken. I was dropping the oldest post out of the search table each time a new one was added. Problem is that doing so made MSSearch do a little more than twice as much work. It's already pretty busy when a new post is added, parsing the words and adding the references to the catalog. When a post gets removed, it appears it's actually more work for MSSearch to find the (potentially thousands of) references where that post number has been associated with that word and remove the reference.

Anyone who has access to the regular Public Msgs search also has access to Advanced Search.

If a future version of SQL Server handles full-text search differently (and from what I've read so far, Yukon won't in the way that's most meaningful to us), I'll be able to go back to keeping the search data in one table rather than segregating it by year.

Don't forget guys: Advanced Search was completely broken last week. The dataset had gotten so large, it was almost always timing out. The only workable solution was to decrease the size of the searched dataset for any given search. And while I was doing that, I incorporated other ways to speed it up, like segregating PM's from public messages so I wouldn't have to include a "message type" text field in the where clause.

PM's are still a single-table deal, though. They'll remain that way for as long as it's possible (searches not timing out consistently), but that might change sometime next year. Currently, about 1.5 million of the messages here are PM's and that's growing rapidly. I haven't taken a read lately on what percent of new messages are private, but it looks large based on how much the PM search table grows each day.

Discover What Traders Are Watching

Explore small cap ideas before they hit the headlines.

Join Today