Sphider-plus

Index only files and documents with defined suffix : If activated, all pages of the site will be searched for links, but only files with suffixes as defined in the docs list will be indexed. For details see chapter Index only files and documents with defined suffix New feature: 1. Perform a WHOIS check for sites waiting for approval in Admin . . .
. . .
option to be activated in Admin backend: Crawler can leave domain during index procedure, but only for canonical links. Only the canonical link will be indexed, but links found there will be ignored. New feature: Obey the 'refresh' meta tags as part of HTML headers. Now following the redirection and delayed indexing. New option: Support UTF-16 . . .
. . .
bit Operating Systems. For details, please notice chapter PDF converter for Linux/UNIX systems New feature: Follow links placed in JavaScript files. Will detect and follow links like document.write(' <a href="new_12.pdf">All news 2012</a> '); Also the complete content of document.write( this text in all rows'); will be indexed and . . .
. . .
charset will be used. New option: Delete duplicate parts of the URL path found in the indexed page URL and the new links. Unfortunately some CMS seem to be unable to build up a correct path for relative links. If activated in Admin backend, these duplicate parts of the path will be deleted from the link URL. Should be activated only, if sites are . . .
. . .
database at the bottom of result listing. To be activated in Admin backend, the count of sites, categories, page links and keywords are displayed. New feature: Automatically deleting invalid URLs from Admin 'Sites' view. Improved 'Add site' function in Admin backend. Now treating URLs with and without 'www' as equal, and excluding them as . . .
. . .
Now holding name and suffix of the banned domains, and no longer the URLs. Improved index procedure Now ignoring links that try to link to the calling URI (self back linking). Improved link detection for relative links, which are to be found in full text. Improved input protection against SQL injections Improved Admin statistics Now providing . . .
. . .
providing also the IP, country code and country name for - Search log - Most popular searches - Most popular page links - Most popular media links Updated GeoIP database, used to provide the IP, CC and country name for the Admin statistics. Now also supporting IPv6 URLs. Support on Windows systems temporary removed for ppt files, as the . . .
. . .
if the URL to be indexed contained blank characters. Bug fixed, which caused invalid URL creation for relative links containing a file name and/or query. Bug fixed in option 'Crawler can leave domain'. Bug fixed in option 'Use list of div ids to ignore the div content during index/re-index'. Bug fixed in option 'Enable to decode entity coded . . .

words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> 4.6 Indexing only parts of a page by <div id='abc'> 4.7 Ignore HTML elements defined by <tagname> . . . . . .
. . .
Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table - Clear Search log - Clear 'Most Popular Page links' log - Clear 'Most Popular Media links' log - Clear . . .
. . .
with ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular links (click counter). - Search . . .
. . .
image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in chapter Introduction , this search engine uses some PHP libraries and extensions. When opening the Settings interface, the existence of these libraries are tested by . . .
. . .
settings you may select the following options: Full: Indexing continues until there are no further (permitted) links to follow. To depth: Indexes to a given depth, where depth means how many clicks away the page can be from the starting page. Depth 0 means that only the starting page is indexed, depth 1 indexes the starting page and all the . . .
. . .
selected to update the database. Spider can leave domain: By default, Sphider never leaves a given domain, so that links from domain.com pointing to domain2.com are not followed. By checking this option Sphider can leave the domain, however in this case its highly advisable to define proper must include / must not include string lists to prevent . . .
. . .
with the same domain name and it also ignores TLD, SLD and www. If e.g. calling from http://www.sphider-plus.eu links like: - http://sphider-plus.eu (without www.) - http://www.info/sphider-plus.eu (additional subdomain) - http://www.sphider-plus.com (different TDL) - http://www.sphider-plus.tec.eu (additional SLD) will be followed if this . . .
. . .
There are 2 different options available in Admin setting to cover this feature. The first one is following all links found during index procedure. The second one is only following the links to other hosts, if the found links are redirected. 2.3 Word stemming Sphider-plus is offering language specific stemming algorithms for 15 languages: . . .
. . .
the readme.pdf documentation. 2.9 Follow Sitemap file To be activated in Admin settings, Sphider-plus will use the links found in sitemap.xml or sitemap.xml.gz files. This significantly increases the speed for index and re-index, because the links will not have to be searched in text part of each page. This option will also force Sphider-plus to . . .

/include/searchfuncs.php /include/search_media.php /include/media_counter.php /include/search_linksphp.php /include/search_media.php /include/common/audio.txt /include/common/image.txt /include/common/suffix.txt /include/common/video.txt /include/images/ all files /include/mediacache/ (new empty folder) /include/textcache/ (new . . .
. . .
during index / re-index procedure the following information will be presented individual for each page: - New linksphp found here - New keywords found here For more details, please notice chapter Error messages and Debug mode New item in Admin / Settings / General Settings: - Enable / Disable MySQL and PHP error messages. It is recommended to . . .
. . .
file changed from "," to " ". As suggested by Ranbir. Improved Admin / Settings section: - Included directory with linksphp to the different Setting blocks. New item in Admin / Settings section: - Backup current configuration settings. Individual files are created with date and timestamp. - Restore configuration settings from former created backup . . .
. . .
/include/categoryfuncs.php /include/commonfuncs.php /include/searchfuncs.php /include/search_linksphp.php /include/suggest.php /include/ajax/ (all files) /include/common/ (all files) /include/js_suggest/ (folder no longer required) /languages/ro-language.php /settings/conf.php /templates/all folder/thisstyle.css Top [ Outdated . . .
. . .
listing. Additional item in Admin settings to select the chronological order of result listing: - 'Most Popular linksphp ' on top. Activating this item, Sphider-plus will present the result listing in order of before learned link attractivity. Defined as those linksphp with the best user acceptance (clicks). Additional items in Statistics overview: . . .
. . .
in Statistics overview: - Queries total - Link clicks total Additional item in Admin / Statistics: - Most Popular linksphp. Presenting the quantity of clicks individual for each link with date and time of last click. Also the latest query before clicking that link is presented. Additional item in Admin / Clean: - Clear 'Most Popular linksphp' log. . . .
. . .
be taken into account when clicking a 'Most popular searches' suggestion. Bug fixed that seduced Sphider to follow linksphp that are placed in HTML comments. Bug fixed that created a wrong weighting calculation for keywords placed - behind a word that did not match 'min_word_length' - behind a 'common' word - first found in full text Bug fixed in . . .
. . .
files ..,/settings/conf.php Attention: Starting with version 1.6, Sphider-plus supports logging of 'Most popular linksphp'. This item requires additional rows in 'linksphp' table of the database. If you update from a former version of Sphider-plus, please run the /admin/install_bestclick.php script. If you upgrade from original Sphider or install . . .
. . .
files cause indexing problems. If 'Follow sitemap.xml' is activated and a valid sitemap was found, the log output linksphp found: 0 - New linksphp: 0 is no longer shown. Because all linksphp are delivered from the sitemap file and new linksphp are not searched during index / re-index. An eventually non-existing log folder will be created automatically . . .

backend. New feature: Index only feeds and ignore all other page content like text and media. Never the less all links will be followed. To be activated in admin backend. New feature: Index Youtube hosted videos. To be activated in admin backend. New feature: If the option 'Search strictly for search results' is not activated in admin settings, . . .
. . .
New feature: Ignore the content of meta tags like , which are placed in body part of the HTML. Never the less all links will be followed. To be activated in admin backend. New feature: Ignore the content inside of noscript tags like <noscript> THIS CONTENT </noscript> , which might be placed in body part of the HTML. To be activated . . .
. . .
of cookies, which might be added to the page content. To be activated in admin backend. New feature: Do not store links and their attributes as keywords. To be activated in admin backend. New feature: Database support for full UNICODE, including astral symbols. Requires MySQL server version 5.5.3 New feature: Compressed transfer on the Internet . . .
. . .
procedure: If 'Spider can leave domain during index procedure' is not activated in admin settings, the external links are not stored as keywords. Improved UP and DOWN buttons in admin 'Settings' menu, and also in result listing. Wrapper added to bypass the PHP bug (error known since PHP v.5.3) gzopen() = gzopen64() and all other gz functions. . . .

log in form. Why? Unable to log into the database 'Configure' menue. Always re-directed to the log in form. Why? Links are not followed during Re-index, only main URL is indexed (option 1). Links are not followed during Re-index, only main URL is indexed (option 2). How to integrate Sphider's search form into existing pages? How to transfer . . .
. . .
1" are displayed. Unable to search for several words like clock, file and system . Why? Indexing stopped after 20 Links, but my site contains more than 650 pages. Don't see the new Links, keywords and thumbnails on my screen during indexing, why? How to fasten the index procedure? Periodical indexing does not work. In the search results I'm . . .
. . .
chapter 2.2 Allow to index other hosts in same domain In case that also foreign domains should be indexed, because Links are redirected to them, it is necessary to enable: 'Spider can leave domain' in Sites view / Options / Edit / Advanced Options individually for each URL Top Why do I get the message 'The search string was not found as part of . . .
. . .
"1"; Replace this row with $db_count = "0"; Afterwards you should be able to enter into the 'Config' submenue. Top Links are not followed during Re-index, only main URL is indexed (option 1). It is not a bug, it is a feature. If 'Follow sitemap.xml' is activated in Admin settings, Links will only be followed if: - 'last modified' date in . . .
. . .
only relevant pages will be indexed, this approach significant reduces the time required for index and re-index. Links are not followed during Re-index, only main URL is indexed (option 2). If a .htaccess file is used in order to redirect requests, or to 'produce' seo friendly link names, it might be helpful to enable the checkbox 'Allow other . . .
. . .
example and with respect to the amount of keyword/link relationships, a database containing 115 sites 5.388 page Links 109.224 keywords, might occupy about 260 MB. There is no chance to overwrite the according provider settings. Top Error message: MySQL failure: Specified key was too long, max. key length is 767 bytes This is an issue with . . .
. . .
MySQL server in one piece. Because the complete full text of each page been indexed, is stored in database (table: Links, column: fulltxt). In order to fix this issue, try the following: On your server in folder /mysql/bin/ find the script my.ini Open this script in your editor and define: max_allowed_packet = 20M Afterwards you need to restart . . .
. . .
also the following solution might solve the problem of too much text transferred to the database table 'Links', column 'fulltxt'. On your server in folder /mysql/bin/ find the script my.ini Open this script in your editor and define: max_allowed_packet = 4M innodb_buffer_pool_size = 32M innodb_log_file_size = 16M Afterwards you need to . . .
. . .
be deleted. Always together with the following OR selector ( for example clock ). Top Indexing stopped after 20 Links, but my site contains more than 650 pages. On a 'Shared Hosting' server each user only gets a small time slice of processor time until the task of the next user will be processed. The time slice for each user will be about some . . .

Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable . . .
. . .
Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot as part of the text result listing. Created during index procedure for . . .
. . .
of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - links holding promoted catchwords on top. 5 different modes of sorting the media results Admin . . .
. . .
. Independent from singular or plural query string. Extensive user statistics Search log, Most popular text links, Most popular media links, User IP, Country code, Host name, Last queried, Top keywords, etc. GDPR support By default, Sphider-plus collects and processes user data in the G eneral D ata P rotection R egulation (GDPR) compliant . . .
. . .
win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All . . .
. . .
and archives Supports compressed (X)HTML, XML and also PDFs, all kind of feeds, frames and iframes in archives. links found in the compressed files are followed. Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. links found in the . . .
. . .
found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be activated separately for Admin backend and User interface. Automatic detection of users preferred dialog language. Admin selectable for self-detection of the preferred language . . .
. . .
<div id='this_value'> and </ as well as <div class='this_value'> and </ will be ignored. However links inside the tags are followed. Multiple and nested divs are attended. The same feature is available for classes in ul and pre tags. Index only parts of a site. < id/class value driven A common list of div id values is used to . . .
. . .
id='this_value'> and </ as well as <div class='this_value'> and </ ; will be indexed, however links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, . . .

Not include' also to prevent erasing of involved URLs" If activated, also erasing of the involved sites and pages (links)) will be prevented. In order to erase all sites and all pages completely, it might become necessary to uncheck this option Improved search form. Now offering separated search buttons for 'text' and 'media' queries, as well as a . . .
. . .
the admin backend, now also all keyword relationships to that site are withdrawn from the database. Site-specific links), category relationships and other dependencies, like registrations in temporary and pending tables, had been already observed before. Improved Admin search function: Searching for 'Sites', the result listing now will present . . .
. . .
Erase, Delete, Pages, Browse and Statistics Improved index procedure for media indexing: No longer accepting dead links). In order to become indexed, the media file must be present. Improved index procedure to speed up indexing. Improved index procedure to cooperate with those servers that do not accept basic authentication strings. Improved . . .

No longer aborting the complete indexation for 'NOHOST' and 'Too many re-directions'. If detected on single links (pages), only the involved links will be bypassed. Improved index and search procedures: Convert all kind of accents and diacritics like á, ç, ê, ì, ü, into their basic vowels Will present the same results for queries with and . . .
. . .
of % 2F in URLs. Instead of using the Apache rewrite module and NE flag, a PHP solution was implemented. So, those links will not break during index procedure. Improved error report for false function of PDF converter during index procedure. Updated bot and harvester list (black IPs) to prevent unwanted queries. Bug fixed: More than 3 . . .

view a database overview is presented like: Database 1 with table prefix 'search1_' contains: 2 sites 9 page links 0 categories 417 keywords 13 media links 8. In order to adopt the Sphider-plus scripts to the individual paths (addressing) of your server, in admin backend open the menu 'Settings' and press any 'Save' button. This will . . .

and if multiple Sitemap files are available, Sphider-plus will process the secondary Sitemaps and extract all links for index / re-index. Also gzip-compressed files (Index Sitemap files as well as the Sitemap files) will be processed. Improved index / re-index procedure: If charset of a site to be indexed is undetectable, because it is not . . .
. . .
used for the involved link. Improved index / re-index procedure: If Sphider-plus is relocated by http 301 or 302, links found at the relocated site will also be followed. For new sites, as per default the spider-depth is now set to 'full'. Improved UTF-8 support: Conversion into UTF-8 charset now is obligatory. Improved index and re-index . . .