Extension talk:SphinxSearch

Archives

Feature requests Search options and features Troubleshooting Prio the Liquid threads switch (May 2012) Resolved and deprecated issues

The talk page now uses liquid threads, all open messages prior to liquid threads have been converted into a talk subpage and can be found here.

It is recommended to use SphinxSearch 0.8.5 and a recent stable release of Sphinx 2.1. Please bear in mind that this extension only handles the communication between MediaWiki and Sphinx and any specific issues related to the search feature (character sets, ability to search with *, search categories, minimum length on search terms etc.) are handled in Sphinx (see sphinx.conf file) and those questions should be redirected to the Sphinx forum.

The development on this extension is done on a voluntary basis and while this forum provides a platform to share experiences and solutions, it is up to its community members to fill in suggestions.

Help

When seeking help and/or support, you might want to consider to mention your system environment (SphinxSearch Extension version, Sphinx version, MW version etc.) otherwise it might be difficult for people to make appropriate recommendations.

For Windows users and related issues see here, for Linux users and related issues see here, and some advice on how to configure a SQLite setup see here.

Previous page history was archived for backup purposes at Extension talk:SphinxSearch/LQT Archive 1 on 2015-07-10.

If anyone, after upgrading to MW 1.34, is finding out that any recently edited pages disappear completely from the index, fear not. There's a fix.

MediaWiki is changing the manner in which content is referenced in the database. To configure Sphinx to correctly index all pages using the new schema, change the following sphinx.conf parameter:

   sql_query  = SELECT page_id, page_title, page_namespace, page_is_redirect, old_id, old_text FROM page, revision, text WHERE rev_id=page_latest AND old_id=rev_text_id

to:

   sql_query = SELECT page_id, page_title, page_namespace, page_is_redirect, old_id, old_text FROM page, slots, content, text WHERE slots.slot_revision_id=page.page_latest AND content.content_id=slots.slot_content_id AND text.old_id=REPLACE(content.content_address, 'tt:', '')

Thank you!

I upgraded mediawiki to 1.34 and SphinxSearch to the version that matches version 1.34. When I run a search I get this error:

[895083e3cfe4edcfab999700] /testwiki/index.php?search=Celia&title=Spezial%3ASuche&go=Seite ArgumentCountError from line 28 of /srv/www/htdocs/testwiki/extensions/SphinxSearch/SphinxMWSearchResult.php: Too few arguments to function SphinxMWSearchResult::getTextSnippet(), 0 passed in /srv/www/htdocs/testwiki/includes/widget/search/FullSearchResultWidget.php on line 65 and exactly 1 expected

When I took a look at SphinxMWSearchResult.php found that getTextSnippet is expecting the parameter terms

       public function getTextSnippet( $terms ) {

However I found this in the 1.34 release notes.

* SearchResult::getTextSnippet( $terms ) the $terms param is being deprecated

  and should no longer be passed. Search engine implemenations should be

  responsible for carrying relevant information needed for highlighting with

  their own SearchResultSet/SearchResult sub-classes.

Is the error caused by a bug or by a configuration issue?

This is a bug in the SphinxSearch extension code. I submitted a pull request here with the necessary changes: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/SphinxSearch/+/587360/

Thank you, Lavamind!

diff -rupN SphinxSearch/SphinxMWSearch.php /opt/SphinxSearch/SphinxMWSearch.php

--- SphinxSearch/SphinxMWSearch.php     2020-07-14 05:23:08.000000000 -0500

+++ /opt/SphinxSearch/SphinxMWSearch.php        2021-03-05 12:51:28.199277717 -0600

@@ -73,7 +73,7 @@ class SphinxMWSearch extends SearchDatab

        }

        public static function infixSearch( $namespaces, $term, $limit, &$results, $offset = 0 ) {

-               $search_engine = new SphinxMWSearch( wfGetDB( DB_REPLICA ) );

+               $search_engine = new SphinxMWSearch( wfGetLB( DB_REPLICA ) );

                $search_engine->namespaces = $namespaces;

                $search_engine->setLimitOffset( $limit, $offset );

                $result_set = $search_engine->searchText( '@page_title: ' . $term . '*' );

diff -rupN SphinxSearch/SphinxMWSearchResult.php /opt/SphinxSearch/SphinxMWSearchResult.php

--- SphinxSearch/SphinxMWSearchResult.php       2020-07-14 05:23:08.000000000 -0500

+++ /opt/SphinxSearch/SphinxMWSearchResult.php  2021-03-05 12:46:22.049271402 -0600

@@ -15,9 +15,10 @@ class SphinxMWSearchResult extends Revis

        public $sphinx_client = null;

-       public function __construct( $row, $sphinx_client ) {

+       public function __construct( $row, $sphinx_client, $terms ) {

                $this->sphinx_client = $sphinx_client;

-               $this->initFromTitle( Title::makeTitle( $row->page_namespace, $row->page_title ) );

+        $this->initFromTitle( Title::makeTitle( $row->page_namespace, $row->page_title ) );

+        $this->terms = $terms;

        }

        /**

@@ -25,7 +26,7 @@ class SphinxMWSearchResult extends Revis

         *

         * @return string highlighted text snippet

         */

-       public function getTextSnippet( $terms ) {

+       public function getTextSnippet( $terms = [] ) {

                global $wgAdvancedSearchHighlighting, $wgSphinxSearchMWHighlighter, $wgSphinxSearch_index;

                $this->initText();

@@ -34,9 +35,9 @@ class SphinxMWSearchResult extends Revis

                if ( $wgSphinxSearchMWHighlighter ) {

                        $h = new SearchHighlighter();

                        if ( $wgAdvancedSearchHighlighting ) {

-                               return $h->highlightText( $this->mText, $terms, $contextlines, $contextchars );

+                               return $h->highlightText( $this->mText, $this->terms, $contextlines, $contextchars );

                        } else {

-                               return $h->highlightSimple( $this->mText, $terms, $contextlines, $contextchars );

+                               return $h->highlightSimple( $this->mText, $this->terms, $contextlines, $contextchars );

                        }

                }

@@ -51,7 +52,7 @@ class SphinxMWSearchResult extends Revis

                $excerpts = $this->sphinx_client->BuildExcerpts(

                        [ $this->mText ],

                        $wgSphinxSearch_index,

-                       implode( ' ', $terms ),

+                       implode( ' ', $this->terms ),

                        $excerpts_opt

                );

diff -rupN SphinxSearch/SphinxMWSearchResultSet.php /opt/SphinxSearch/SphinxMWSearchResultSet.php

--- SphinxSearch/SphinxMWSearchResultSet.php    2020-07-14 05:23:08.000000000 -0500

+++ /opt/SphinxSearch/SphinxMWSearchResultSet.php       2021-03-05 12:47:39.304272996 -0600

@@ -233,7 +233,7 @@ class SphinxMWSearchResultSet extends Se

                if ( isset( $this->mResultSet[$this->mNdx] ) ) {

                        $row = $this->mResultSet[$this->mNdx];

                        ++$this->mNdx;

-                       return new SphinxMWSearchResult( $row, $this->sphinx_client );

+                       return new SphinxMWSearchResult( $row, $this->sphinx_client, $this->mTerms );

                } else {

                        return false;

                }

Sphinx Search can index HTML files (according to this: Indexing html with Sphinx without complex scripts). Is this possible with this extension?

Line 76 of SphinxMWSearch.php is incorrectly coded.

SearchDatabase class requires an ILoadBalance construct

Without you get error:

SearchDatabase::__construct() must implement interface Wikimedia\\Rdbms\\ILoadBalancer,

        instance of Wikimedia\\Rdbms\\MaintainableDBConnRef given,

        called in /var/www/html/mediawiki/extensions/SphinxSearch/SphinxMWSearch.php on line 76

To fix, change line 76 from wfGetDB( DB_REPLICA ) to wfGetLB( DB_REPLICA )

I am using SphinxSearch 0.9.1 for MediaWiki 1.31 LTS. I am trying to customize the advanced search (searching using namespaces) that is included in my Special:Search page. I have found the following manual for altering the default namespace: Manual:$wgNamespacesToBeSearchedDefault. However, the changes I make on my LocalSettings.php does not seem to be reflecting on the Special:Search page.

Specifically, I have included the following code:

$wgNamespacesToBeSearchedDefault = [NS_HELP => false, ];

in LocalSettings.php to see if it would disable the "Help" namespace from the advanced search, but there was no change to the page.

Is there something I am doing wrong? Please help. I am new to MediaWiki and php in general, hence I understand that I may be missing a big point. Thank you!

Although the download page does offer a version for 1.34, at least in my experience that throws errors after installation, for example "[728f57963e6ba6317d0fee1a] 2020-01-30 15:54:50: Fatal exception of type "ArgumentCountError". To be fair, the page we're discussing here only shows compatibility up through 1.33. Is an update something we should expect soon?

I am new to media wiki and for that matter to any web application development. I was exploring Mediawiki and have installed "MediaWiki 1.18.2" and have attempted to integrate Sphinx search. I have used "Sphinx 2.0.4-release".

The problem I am facing is that I am unable to search through the contents of the pages or the documents embedded in the MediWiki. It is only able to search the Title of the Pages. This is the same irrespective of the "GO" button or the "SEARCH" button. Whereas Through the command in the Sphinx Search, I am able to find all the text. Can someone please help me in debugging this.

The documentation is quite extensive (compared with other extensions), and it might be that you need to do the setup more than ones to fully understand the implication in how searchd, indexer and SphinxSearch is working together. If you follow the steps outlined in the documentation you should be able get the extension working but it will be hard for anyone to give any guidance as long their are no particular problems to solve.

For example did you do "Step 4 - Test Out Sphinx" to ensure that the infrastructure is setup correctly otherwise if this step fails something is wrong in your configuration and searching within MediaWiki will definitely not work as advertised.

In order to use Sphinx in MediaWiki you have register the extension, see "Step 9 - Extension Installation - Local Settings"

You should also consult the Extension:SphinxSearch/Troubleshooting page.

After migraging to MediaWiki 1.19.0 SphinxSearch 0.85 does not work on the Spechial:Search page anymore (no results). The sidebar Search works.

After logging the searchd with --console, you found a message "query error: no such filter attribute 'page_is_redirect'" in the console. I searched this text and found it in 'SphinxMWSearch.php'. I change the line

$cl->SetFilter( 'page_is_redirect', array( 0 ) );

to

// $cl->SetFilter( 'page_is_redirect', array( 0 ) );

and everything works (for me).

I also change the method 'userHighlightPrefs' to

	public static function userHighlightPrefs() {return array( 2, 75 ); }

because the the definition changed (and I got warning when I activated "$wgShowExceptionDetails" in LocalSettings.php).

I hope this helps others fixing the same problems.

Perhaps page_is_redirect is missing in the sphinx.conf? It did not use to be there by default, but in more recent versions you should see it in at least three places in your sphinx.conf, in both sql_query settings, and in one sql_attr_uint setting. After you change the config file, you need to reindex the wiki.

Hello Svemir,

thanx, that was the solution.

During our tests with MW 1.19 / SphinxSearch 0.8.5 we haven't seen any problems and therefore are able to run SphinxSearch 0.8.5 without any changes on MW 1.19.

I got tripped up by the use of 'spinxconf' instead of 'sphinxconf' as an option to SphinxSearch_setup.php. The following patch may avoid minor confusion for new users:

--- a/SphinxSearch/SphinxSearch_setup.php   2012-06-04 14:08:10.000000000 -0700
+++ b/SphinxSearch/SphinxSearch_setup.php   2012-06-05 12:59:57.289298038 -0700
@@ -24,7 +24,7 @@
        
        /* Override parameters setup becuase we do not need some of the default ones */
        protected function addDefaultParams() {
-               $this->addOption( 'spinxconf', 'Location of Sphinx configuration file', true, true );
+               $this->addOption( 'sphinxconf', 'Location of Sphinx configuration file', true, true );
                $this->addOption( 'indexer', 'Full path to Sphinx indexer if not in the path', false, true );
                $this->addOption( 'useindex', 'Sphinx index to use (defaults to wiki_main)', false, true );
                $this->addOption( 'maxwords', 'Maximum number of words to extract (defaults to 10000)', false, true 
@@ -36,7 +36,7 @@
                $max_words = intval( $this->getOption( 'maxwords', 10000 ) );
                $indexer = wfEscapeShellArg( $this->getOption( 'indexer', 'indexer' ) );
                $index = wfEscapeShellArg( $this->getOption( 'useindex', 'wiki_main' ) );
-               $conf = wfEscapeShellArg( $this->getOption( 'spinxconf' ) );
+               $conf = wfEscapeShellArg( $this->getOption( 'sphinxconf' ) );
 
                $cmd = "$indexer  --config $conf $index --buildstops sphinx.dic $max_words";
                $this->output( wfShellExec( $cmd, $retval ) );

Thank you for the fix. MW and extensions are moving from SVN to Git - I will apply and commit your change once all that is done.

It seems like its going it be a while before this extension is moved to Git. I made the change in SVN now (revision 115486)