Forums

Hi admins,

Looks like the search index is corrupted. I tried today to search for some old posts and all I got was "No results"...

Try for instance to search for "sms power radio". No results.

Even for "sms" there are no results !

The only thing that got me results was "master system" oO.

Really odd huh ?

Niloct.

While it is pretty bad, it's not that bad - it just does badly when you give it search words smaller than 4 letters. Try just "radio", for example.

Try looking for "new games"... Lots of results.

Then add "sms". Search for "new games sms". No results...

The 1st search has a "bad" keyword and it still return results.

Nevertheless, I don´t like this, ´cause when one has to look using special (tiny) keywords one will never know if there are indeed no threads in the forums.

To be honest... searches suck and I don't know exactly why. You're probably better off with Google anyway.

Maxim wrote

To be honest... searches suck and I don't know exactly why. You're probably better off with Google anyway.

Would making the search function here google enhanced help??

The advantage of our search is that you can arrange the results by thread, you get word highlighting, and it's all inside the forums so it matches the look and feel. The advantage of Google is that it is less sucky. I might look into why "sms" kills searches.

At a geuss its probably using the mysql fulltext search function which has some annoying caveats, following probably best describes them:

"FULLTEXT indexes are most often used to search natural language text, such as through newspaper articles, web page contents and so on. For this reason MySQL has added a number of features to assist this kind of searching. MySQL does not index any words less than or equal to 3 characters in length, nor does it index any words that appear in more than 50% of the rows. This means that if your table contains 2 or less rows, a search on a FULLTEXT index will never return anything. In future, MySQL will make this behavior more flexible, but for now it should suit most natural language uses."

You can infact change the behavior via mysql variables.

It's almost like that. phpBB doesn;t use fulltext searching because it has to be portable across a bunch of database servers. Instead it has a table of search words, and a join table for the many-to-many link between words and posts. Quite why it does this is a bit of a mystery, presumably someone thought it was a good idea.

There's a list of common words that it won't bother putting in the search word table (the, and, etc), and beyond that checks which words appear in more than 40% of all posts and flags them as "common". We (apparently) have eight:

forum smspower html http org com sms music

All links for these words are thus removed and adding any of them to a search will make it return no results. It would be better if the code would instead remove them from the search terms, or at least tell you that you'd used a banned word. If I have time I'll try to change it.

[Admin] Possible mitigations: 1. rebuild the search index, especially since it probably never ran on the imported posts. 2. switch to a MySQL fulltext search, which has certain advantages, eg. it supports quoted phrase searches, might be faster, might use less DB resources.

I mean, just search "master system" and flip to the last page. You'll find old posts there, but don't try to bring them back.

Author	Message
Niloct Joined: 25 Dec 2005 Posts: 607 Location: São Paulo - Brazil	[Forums] - Searching Posted: Sat Jan 26, 2008 5:12 pm
	Hi admins, Looks like the search index is corrupted. I tried today to search for some old posts and all I got was "No results"... Try for instance to search for "sms power radio". No results. Even for "sms" there are no results ! The only thing that got me results was "master system" oO. Really odd huh ? Niloct.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14735 Location: London	Posted: Sat Jan 26, 2008 8:18 pm
	While it is pretty bad, it's not that bad - it just does badly when you give it search words smaller than 4 letters. Try just "radio", for example.

Niloct Joined: 25 Dec 2005 Posts: 607 Location: São Paulo - Brazil	Posted: Sat Jan 26, 2008 8:27 pm
	Try looking for "new games"... Lots of results. Then add "sms". Search for "new games sms". No results... The 1st search has a "bad" keyword and it still return results. Nevertheless, I don´t like this, ´cause when one has to look using special (tiny) keywords one will never know if there are indeed no threads in the forums.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14735 Location: London	Posted: Sun Jan 27, 2008 9:18 am
	To be honest... searches suck and I don't know exactly why. You're probably better off with Google anyway.

Jacko Joined: 01 Feb 2004 Posts: 1464 Location: Sunny ol Tamworth, New England NSW AU	Posted: Sun Jan 27, 2008 9:21 am
	Maxim wrote To be honest... searches suck and I don't know exactly why. You're probably better off with Google anyway. Would making the search function here google enhanced help??

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14735 Location: London	Posted: Sun Jan 27, 2008 12:25 pm
	The advantage of our search is that you can arrange the results by thread, you get word highlighting, and it's all inside the forums so it matches the look and feel. The advantage of Google is that it is less sucky. I might look into why "sms" kills searches.

thatawesomeguy Joined: 25 Jul 2007 Posts: 729 Location: Melbourne, Australia	Posted: Mon Jan 28, 2008 3:29 pm
	At a geuss its probably using the mysql fulltext search function which has some annoying caveats, following probably best describes them: "FULLTEXT indexes are most often used to search natural language text, such as through newspaper articles, web page contents and so on. For this reason MySQL has added a number of features to assist this kind of searching. MySQL does not index any words less than or equal to 3 characters in length, nor does it index any words that appear in more than 50% of the rows. This means that if your table contains 2 or less rows, a search on a FULLTEXT index will never return anything. In future, MySQL will make this behavior more flexible, but for now it should suit most natural language uses." You can infact change the behavior via mysql variables.

Maxim Site Admin Joined: 19 Oct 1999 Posts: 14735 Location: London	Posted: Mon Jan 28, 2008 5:16 pm
	It's almost like that. phpBB doesn;t use fulltext searching because it has to be portable across a bunch of database servers. Instead it has a table of search words, and a join table for the many-to-many link between words and posts. Quite why it does this is a bit of a mystery, presumably someone thought it was a good idea. There's a list of common words that it won't bother putting in the search word table (the, and, etc), and beyond that checks which words appear in more than 40% of all posts and flags them as "common". We (apparently) have eight: forum smspower html http org com sms music All links for these words are thus removed and adding any of them to a search will make it return no results. It would be better if the code would instead remove them from the search terms, or at least tell you that you'd used a banned word. If I have time I'll try to change it. [Admin] Possible mitigations: 1. rebuild the search index, especially since it probably never ran on the imported posts. 2. switch to a MySQL fulltext search, which has certain advantages, eg. it supports quoted phrase searches, might be faster, might use less DB resources.

aqua_hedgehog Joined: 17 Apr 2007 Posts: 120 Location: Anchorage, Alaska	Posted: Mon Feb 04, 2008 8:35 am
	I mean, just search "master system" and flip to the last page. You'll find old posts there, but don't try to bring them back.

Forums

View topic - [Forums] - Searching