Sphider-plus

For more details, please notice chapter Result cache for text and media queries Enlarged Admin statistics. In table 'Search Log' the following items are additionally presented: - User IP - Users country code - Users hostname New item in Admin Settings (Section: Index Log Settings): Suppress browser output of logging data during index / . . .
. . .
is started, so that this adaptation is valid only for the current session. New feature in Admin statistics. In table 'Image functions', details about the installed GD library as part of the PHP environment will be presented. New feature in Admin 'Clean' section: Clean text and media cache (separate items). Additionally count of results in . . .
. . .
/include/textcache/ (new empty folder) /languages/ all files Attention: This release requires additional database tables and additional table rows in already existing tables. If you update from a former version of Sphider-plus, please run the /admin/install_v.1.8.php script. If you upgrade from original Sphider or install from scratch, you don't . . .
. . .
to be indexed can't be detected. Bug fixed that prevented correct link to search result. Additional translation table to convert upper to lower case characters for Cyrillic charset. Updated Russian language file, thanks to vipraskrutka. Version 1.7 Release date: 20. November 2008 New item in Admin / Settings / General Settings: - Enable Debug . . .
. . .
count of query letters in order to get a suggestion. - Show / Hide the amount of found keywords in suggestion table. New capability to prepare language specific common files. If multilingual sites, or sites with different languages, are to be indexed, this feature improves overview. Common words to be ignored during index / re-index . . .
. . .
the root folder of the Sphider-plus installation. If the charset of a page to be indexed / re-indexed is not detectable, the home charset as defined in Admin settings is used. Improved search function for non-Latin symbols. Search function enabled for queries containing an apostrophe. Included query input protection against Directory Traversals. . . .
. . .
version 1.6, Sphider-plus supports logging of 'Most popular links'. This item requires additional rows in 'links' table of the database. If you update from a former version of Sphider-plus, please run the /admin/install_bestclick.php script. If you upgrade from original Sphider or install from scratch, you don't need to run this script. Its . . .
. . .
suggestions are presented also for queries with accented letters. Enable real-time output of logging data. Selectable in Admin setting together with the update interval (1 - 10 seconds). In order to prevent performance problems and memory overflow for large amount of URLs, Sphider-plus may clean resources during index / re-index. Selectable in . . .
. . .
supports real-time output of logging info during index / re-index procedure. This item requires an additional table for the database. If you update from a former version of Sphider-plus, please run the /admin/install_reallog.php script. If you upgrade from original Sphider or install from scratch, you don't need to run this script. Its . . .

It includes extensive regex rules to tags like XSS, SQLI, RCE, LFI, DT, CSRF, LDAP Injections, and DoS. Admin selectable, the IDS will block further user input, create a log-file, present a warning message, or even block any traffic of IP’s known to be evil. For details see chapter: Intrusion Detection System (IDS) New feature Index only links . . .
. . .
are available. New feature in Admin settings: Add new domains found during index procedure to 'Approve Sites' table. To be activated in section ‘General Settings’, this option is available for those sites, having activated ‘Spider can leave domain’ in their options. New feature in Admin sites menu: Index all the suspended. Will continue the . . .
. . .
for all erase functions. Will reset all 'Search' statistics in Admin backend, as well as the 'Most popular search' table at the bottom of result listing New feature in Admin backend: When opening the Admin interface, a warning message will be presented about new suggested sites, waiting for approval. Working independent from the currently . . .
. . .
/admin/admin_footer.php /admin/auth.php /admin/configset.php /admin/index_media.php /admin/install_tables.php /admin/spider.php /admin/spiderfuncs.php /admin/url_backup.php /include/commonfuncs.php /include/ids_handler.php /include/search_links.php /include/searchfuncs.php /include/search_media.php /include/suggest.php . . .
. . .
/templates/Sphider-plus/thisstyle.css Attention: This version requires an updated set of database tables. In order to create the new set of tables, run the 'Install all tables' for all databases in 'Database Management / Configure' menu. Top . . .

thumbnails are no longer stored in a sub folder of the Sphider-plus installation, but now are stored in database table "media" in field "thumbnail". Improved media search: AND, OR and TOLERANT modes are now selectable for media search, while the PHRASE mode will be interpreted as an AND search. Improved media search: Henceforward file name as . . .
. . .
also be used for suggested sites (addurl form). New option in Admin 'Clear' menu: Clear all entries in 'Addurl' table. New option in Admin 'Clear' menu: Clear all entries in 'Banned' table. Improved option: Ignoring parts of a page defined by <;div id='abc'>; now is working alternately also for <;div class='abc'>; Besides the . . .
. . .
"blank" characters. Improved "Erase & Re-index all" function. Now deleting also the "Pending" and "Temp" tables. Support for Greek language totally rewritten. Now accepting Latin characters for old and new Greek transcription. For details see chapter: "Greek language support" Improved parser for RSS v.2.0 feeds. Bug fixed in index . . .
. . .
/admin/configset.php /admin/db_copy.php /admin/db_main.php /admin/index_media.php /admin/install_tables.php /admin/messages.php /admin/real_log.php /admin/spider.php /admin/spiderfuncs.php /converter/feed_parser.php /include/click_counter.php /include/commonfuncs.php /include/make_captcha.php /include/media_counter.php . . .
. . .
results.html /templates/html/140_image-results.html Attention: This version requires an updated set of database tables. It is strongly recommended to follow the instructions as described in chapter: "Updating from 2.x to 2.y" for the actual version 2.8 Top . . .

2. Create at minimum one database as part of the MySQL server, which will hold the Sphider-plus data tabless. Collation of the SQL server connection must be defined to utf8mb4_unicode_ci already before creating any database. In order to get full UNICODE support, the collation of Sphider-plus databases also must be set to . . .
. . .
At minimum one database has to be declared for: Name of database User name Password Database host Prefix for tabless as defined before in step2 externally, when you created the database. Additionally you need to add a free selectables name as tables_prefix. This tables prefix is obligatory. A blank character as prefix is not enough. Minimum . . .
. . .
the 'Save' button will assign Sphider-plus to these database definitions. Never the less, the warning message 'tabless are not installed for database x' will remain in the database settings overview. Now you should get the first (green) message like Database 1 settings are okay. tabless are not installed for database 1 All further steps for . . .
. . .
like Database 1 settings are okay. tabless are not installed for database 1 All further steps for installing the tabless for database1 and activating the database for Admin, Search User and Suggest User remain blocked, until you do not get this part of step 5 to work. The 'Install all tabless for database x' is an independent procedure, which has . . .
. . .
- Import URL list - Index At the bottom of 'Sites' view a database overview is presented like: Database 1 with tables prefix 'search1_' contains: 2 sites 9 page links 0 categories 417 keywords 13 media links 8. In order to adopt the Sphider-plus scripts to the individual paths (addressing) of your server, in admin backend open the menu . . .

New method implemented, when manually aborting the index procedure. Now obligatory a repair, optimize and table flush will be performed. Worked out also for multi threaded indexing. Thumbnails, created of the indexed URLs, will now be taken from top of the page; no longer centred. UTF-8 charset ensured for data transfer from/to MySQL . . .
. . .
usage in sub folder /xml/stored/ To be deleted in Admin backend in 'Clear' menu. Improved support for multiple table sets in one database. If the option 'Do not index the full text' is activated in Settings menu, the 'Required number of words in a page in order to get indexed' is obligatory set to zero Bug fixed in option: Show 'Description' . . .
. . .
/admin/db_common.php /admin/db_copy.php /admin/configset.php /admin/db_main.php /admin/install_tables.php /admin/messages.php /admin/real_get.php /admin/real_log.php /admin/settings_backup.php /admin/spider.php /admin/spiderfuncs.php /include/click_counter.php /include/commonfuncs.php /include/idna_converter.php . . .

Automatically adapting the size of search form, result listing and addurl form to display size of computers, tabletsts, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also indexed and herewith become searchable. Thumbnails for all . . .
. . .
configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple tablets sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of tabletss. Result cache Extremely reduced response time for . . .
. . .
Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectablets intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking . . .
. . .
might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitablets level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or bulk update of database. . . .
. . .
of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectablets, this feature will present a web shot as part of the text result listing. Created during index procedure for all URLs, which had been indexed on the Internet. Prevent indexing of known malware and pishing pages This feature is . . .
. . .
pages, suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectablets: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectablets: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To . . .
. . .
Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectablets; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration . . .
. . .
for Admin backend and User interface. Automatic detection of users preferred dialog language. Admin selectablets for self-detection of the preferred language of the search engine user. Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectablets for: Arabic, Bengali, Bulgarian, Catalan, . . .

Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table - Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail . . .
. . .
- Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases . . .
. . .
listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates . . .
. . .
below for details . Word stemming. See below for details . 2.2 Allow other hosts in same domain This Admin selectable option allows to index other hosts with the same domain name and it also ignores TLD, SLD and www. If e.g. calling from http://www.sphider-plus.eu links like: - http://sphider-plus.eu (without www.) - . . .
. . .
or site specific, started periodically at a defined time interval. In Admin backend the time interval is selectable to 3 hours, 12 hours, 1 day, 1 week or 1 month. Also the count of periodically performed re-indexing procedures could be defined in Admin backend. Once started, the Re-index procedures will silently work in the background . . .
. . .
automatically set to '1'. While invoking the option 'Re-index only preferred sites', the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed. 2.6 Multithreaded indexing The Admin setting: Define number of threads allowed for index procedures (max. 10) . . .
. . .
for command line operation is presented below. 2.7 Create thumbnails during index procedure This Admin selectable option is supplied by a web service with an individual key, which needs to be signed up by the Sphider-plus admin. The web shots will be presented as thumbnails with the size of 130 X 174 pixels as part of the text result . . .
. . .
option will also force Sphider-plus to re-index only links that are: - New and not jet known in Sphider's link table - Links with a 'last modified' date, which is newer than Sphider's 'last indexed' date in database. Sitemaps are always expected in the root folder of the site to be indexed and must be named sitemap.xml or sitemap.xml.gz If . . .
. . .
by once preparing the database with the command php spider.php <-preall> This will reset all 'Last indexed' tables to '0000', but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing . . .

independent from Admin login. Integrated availability check for all databases and their release relevant table structure. Individual for each database: - Backup and restore - Copy / Move from each database to each other database 32 MByte query cache for MySQL database. - To be activated in Admin settings. - Status of cache is observable . . .

word is stored as keyword, all 4 words become searchable. Alternatively the separation only at hyphens is selectable in Admin settings. New feature: Index the "Description" Meta tag in HTML header. To be activated in Admin settings. New feature: Index of media files enabled for those servers that do not offer all PHP functions for remote . . .
. . .
in the compressed files will be followed. 15 language specific stemming algorithms implemented. Individually selectable for: Bulgarian, Chinese, Czech, Dutch, English, Finnish, French, German, Greek, Hungarian, Italian, Portuguese, Russian, Spanish and Swedish. For details see chapter Word stemming More details in documentation chapter: Word . . .
. . .
will be presented on top of the Settings menu. Improved database 'Activate / Disable' menu: If multiple sets of tables are available, because they have been created for a database before, you will be able to activate any of these table sets by selecting the corresponding prefix. New Admin setting: Define directory for templates (relative to . . .
. . .
search.php /admin/admin.php /admin/configset.php /admin/db_config.php /admin/index_media.php /admin/install_tables.php /admin/messages.php /admin/real-log.php /admin/spider.php /admin/spiderfuncs.php /admin/url_backup.php /admin/getid3/module.audio-video.asf.php /converter/catdoc.script /converter/xls2csv.exe (not required any longer) . . .

'Use list of div ids to ignore the complete div content during index/re-index'. Bug fixed in 'Most Popular search' table at the bottom of result listing. Bug fixed while indexing <! sphider_noindex > Bug fixed in result listing, while sorted 'Like Google'. Debug info removed from index procedure. MySQLi error messages now are correctly . . .