# Date: 2001-07-14 ========================= fantomas multiBlocker(TM) ========================= ************************************ * ONLINE HELP IS AVAILABLE AFTER * * SUCCESSFULL INSTALLATION BY * * CLICKING ON THE * * FANTOMAS LOGO! * ************************************ SYSTEM REQUIREMENTS INSTALLATION - UNIX - NT - THE ".htaccess" FILE UNINSTALLING THE PROGRAM WORKING WITH fantomas multiBlocker(TM) ERROR HANDLING KNOWN ISSUES UPDATES + PROGRAM CHRONOLOGY CONTACT + SUPPORT ====================================================================== SYSTEM REQUIREMENTS ------------------- Language ------- Perl 5 UNIX ---- The Unix system requires an installed web server. Execution of CGI scripts and Server Side Includes (SSI) must be enabled. A directory for execution of CGI scripts must be existent. Usually, this will be directoy /cgi-bin/. Tested under: SuSE LINUX with Apache Red Hat Linux with Apache BSDI Unix with Apache NT -- The NT system requires an installed web server. Execution of CGI scripts must be enabled. A directory for execution of CGI scripts must be existent. Usually, this will be directoy \cgi-bin\. Tested under: NT 4.0 with Apache (Win32) Browser ------- Script is called and executed via web browser. You will currently achieve best results under MS Internet Explorer 5+. Netscape 4.7 may require adjustment of font size. Tested under: IE5+, Netscape 4.7, Netscape 6.0, Opera 5.12 ====================================================================== INSTALLATION ------------ The following files are included: multiblocker-e.cgi --- (program script) MBLIBE.pm --- (script routines) hefoo-e.fan --- (HTML header and footer) fantomas.gif --- (logo/graphics file) varparm_mb.txt --- (empty file to store current working parms) .htaccess --- (allows Server Side Includes for .html files) hostlist.txt --- (Host list) iplist.txt --- (IP list) ualist.txt --- (UserAgent list) mbhelp.txt --- (documentation in TXT format) <--- THIS FILE YOU ARE READING! fa_license-e.txt --- (License Agreement And Terms Of Usage - PLEASE READ!) ----------------------------------- ADJUSTMENTS IN FILE "multiblocker-e.cgi" (please edit in ASCII or plain text editor like Notepad etc.) ----------------------------------- UNIX ---- * Please check path to location of Perl. The default path in the script is "/usr/bin/perl". If you don't know this path, you can check it out under telnet by entering Unix command "whereis perl". You may have to adjust the first line in the script "multiblocker-e.cgi" accordingly. * The variables in the script "multiblocker-e.cgi": "$admin_dir", "$parms_file" and "$wraplist" may optionally be adjusted to your requirements. A comprehensive description of these variables can be found below in chapter "WORKING WITH fantomas multiBlocker(TM)". * If you change variable "$parms_file", you will have to rename the included file "varparm_mb.txt" accordingly. * The script "multiblocker-e.cgi", the files "MBLIBE.pm", "hefoo-e.fan", "fantomas.gif" and the help file "mbhelp.txt" must be copied into the Unix server's CGI directory. * The CGI directory must be endowed with the following permissions: "chmod 755" [drwxr-xr-x] * Next, create the directory defined as variable "$admin_dir" with the following permissions: "chmod 777" [drwxrwxrwx](Default name is "admin".) * Now, copy file "varparm_mb.txt" (default) or whatever file you specified under "$parms_file" and the list files "hostlist.txt", "iplist.txt" and "ualist.txt" into this directory. * When uploading via FTP, make sure to transfer ALL files in ASCII mode (including ".fan" and ".pm" files!) EXCEPTION: the graphics file "fantomas.gif" which must be transferred in BINARY or AUTOMATIC mode. * Required file permissions: multiblocker-e.cgi: "chmod 755" [-rwxr-xr-x] MBLIBE.pm: "chmod 444" [-r--r--r--] hefoo-e.fan: "chmod 444" [-r--r--r--] fantomas.gif: "chmod 444" [-r--r--r--] varparm_mb.txt: "chmod 666" [-rw-rw-rw-] hostlist.txt: "chmod 666" [-rw-rw-rw-] iplist.txt: "chmod 666" [-rw-rw-rw-] ualist.txt: "chmod 666" [-rw-rw-rw-] mbhelp.txt: "chmod 444" [-r--r--r--] NT -- * Please check path to location of Perl. You may have to adjust the first line in the script. * The variables in the script "multiblocker-e.cgi": "$admin_dir", "$parms_file" and "$wraplist" may optionally be adjusted to your requirements. A comprehensive description of these variables can be found below in chapter "WORKING WITH fantomas multiBlocker(TM)". * If you change variable "$parms_file", you will have to rename the included file "varparm_mb.txt" accordingly. * The script "multiblocker-e.cgi", the files "MBLIBE.pm", "hefoo-e.fan", "fantomas.gif" and the help file "mbhelp.txt" must be copied into the NT server's CGI directory. * Next, create the directory defined as variable $admin_dir". (Default name is "admin".) * Now, copy file "varparm_mb.txt" (default) or whatever file you specified under "$parms_file" and the list files "hostlist.txt", "iplist.txt" and "ualist.txt" into this directory. * When uploading via FTP, make sure to transfer ALL files in ASCII mode (including ".fan" and ".pm" files!) EXCEPTION: the graphics file "fantomas.gif" which must be transferred in BINARY or AUTOMATIC mode. * Checking the file permissions: the following files must NOT be restricted by attribute "Read-only": varparm_mb.txt hostlist.txt iplist.txt ualist.txt The ".htaccess" file -------------------- If Server Sides Includes have not been implemented for ".html" files, you can activate this option by uploading a customized ".htaccess" file. Note: this applies only to web servers configured for usage of customized ".htaccess" files! The file ".htaccess" should include as a minimum the following entries: Options +Includes AddType text/x-server-parsed-html .html (A sample ".htaccess" file featuring these entries is included with our package. If you wish to make use of the included ".htaccess" file, please take care not to overwrite any existing file! Please be very careful in this matter: many web servers won't show an existing ".htaccess" file (or any other file beginning with a period/dot "." for that matter) in your FTP window! If in doubt, you will either have to access your account by telnet and take it from there, or consult your server administrator. Should you have a ".htaccess" file already installed on your web server, edit it instead to include the entries outlined above.) Upload and file permissions --------------------------- After preparing the ".htaccess" file, you may upload the file in customary manner by FTP. Warning: ".htaccess" - and all other files for that matter - must be uploaded ONLY IN "ASCII MODE", i.e. NOT in Binary Mode! Experience shows that appr. 90% of all installation problems concerning CGI/Perl programs are based on uploading CGI/Perl script either in Binary or "Automatic" mode, which effectively makes them non-executable! .htaccess: "chmod 644" [-rw-r--r--] ====================================================================== UNINSTALLING THE PROGRAM ------------------------ For complete uninstall, delete the following: multiblocker-e.cgi MBLIBE.pm hefoo-e.fan fantomas.gif varparm_mb.txt (default) or whatever file you specified under "$parms_file" hostlist.txt iplist.txt ualist.txt mbhelp.txt The directory "admin" or whatever directory you defined under "$admin_dir" including contents. ====================================================================== WORKING WITH fantomas multiBlocker(TM) -------------------------------------- Program Description ------------------- The fantomas multiBlocker(TM) is a script which allows you to block predefined Hosts, IPs and UserAgents from accessing your web site. If a visitor's or searchbot's data is found to match with the predefined value, the whole server process will be put to "sleep". The "Sleep Phase" is configurable. Normally a spider (searchbot, crawler) will only wait for a given amount of time before canceling the process (timeout). Our extensive analysis of access stats has revealed that e.g. the spider belonging to "www.cyveillance.com" will wait for a full 20 minutes before effecting a timeout. In this example only appr. 100 Bytes will be transferred. The "sleep" method is useful for saving bandwidth when fending off undesirable spiders. Moreover, during the sleep phase your system resources will not be put to any strain. The Control Center Script "multiblocker-e.cgi" offers two functions: a) Generation of the Perl blocking script. b) Maintenance of list defining Hosts, IPs and/or UserAgents you wish to deny access. Detailled Description --------------------- a) Generation of the Perl blocking script Via the Control Center Script "multiblocker-e.cgi" you can generate a blocking script (standard name is: mb.cgi). To execute this script, add a call in the header of every HTML web page you want to protect. (This procedure is explained in more detail further below.) Now, the script will be activated whenever one of these prepared HTML pages are hit by a targeted visitor. Visitor/searchbot will be checked against the predefined values (Host, IP, UserAgent) and will be blocked from access if the predefined criteria are met. In this case, the visitor will be put to "sleep". The HTML Code required to trigger the script will be generated together with the Perl script proper and can then be transferred by "cut and paste" into the HTML pages you wish to protect. --------------------------------- Customization of script variables --------------------------------- The following variables may optionally be customized in script "multiblocker-e.cgi": * $admin_dir This variable defines the directory where the parms file and the blocking lists for Hosts, IPs and UserAgents shall reside. Default file name is "admin". * $parms_file This variable defines the file name in which the current parameters are stored. Our package includes an empty parms file named "varparm_mb.txt". * $wraplist This variable defines whether the blocking lists shall be word wrapped (default is: "off"). It may alternatively be set to "virtual", in which case the blocking lists will be displayed in word wrap mode. --------------------- Configuration Options --------------------- By calling the Control Center Script from your web browser you can configure your options. E.g.: "http://www.YourDomain.com/cgi-bin/multiblocker-e.cgi". All configurations are in relation of the script you wish to generate. First, select your blockage option(s). Required minimum is one, else the program will generate an error message if you try to generate a script. (For maintenance of blocking lists see below.) 1. Option: "block by Host" Visitors will be blocked in dependency on their respective Host. Example Hosts entries: cr000.digital-integrity.com cr001.digital-integrity.com You may also block a complete host range by entering as Host: digital-integrity.com To block a complete Top Level Domain (i.e. the national domains belonging to a whole country), you might enter: .ro (In this example, all visitors - human and robots alike - attempting to access your site via a server hosting a Romanian domain will be blocked.) This feature is helpful for webmasters worried about credit card and chargeback fraud originating in more than statistical average from established foreign domains. The entered string is checked for compliance from RIGHT to LEFT. Thus, while "www.domain.ro" will be blocked, "ro.domain.com" will not be. 2. Option: "block by IP" Visitors will be blocked in dependency on their respective IP. Example IP entries: 62.200.157.32 198.138.63.17 You may also block a complete IP range by entering as IP: 198.138.63. (In this example, all IPs between 198.138.63.1 and 198.138.63.255 will be denied access.) The entered string is checked for compliance from LEFT to RIGHT. Thus, in the example above, while "198.138.63.78" will be blocked, the IP "78.198.138.63" will not be. 3. Option: "block by UserAgent" Visitors will be blocked in dependency on their respective UserAgent. Example UserAgent entries: EmailCollector EmailSiphon Zeus 94664 Webster Pro V2.9 Win32 You may also block a complete UserAgent range by entering as UserAgent: Zeus The entered string is checked for compliance from LEFT to RIGHT. Thus, in the example above, while "Zeus 94664 Webster Pro V2.9 Win32" will be blocked, UserAgent "Web Zeus" will not be. This feature is especially useful to fend off email harvester or extractor bots will scour your site for valid email address to abuse for spam mail. --------------- VARIABLE FIELDS --------------- After selecting a minimum of one blocking option, the following fields will have to be filled out. 4. "Path to Perl 5:" Enter the path to Perl 5 on your system here. This entry will constitute the first line of the generated script. 5. "Name of admin directory:" Define the directory in which the log file shall reside. You may enter either an absolute or a relative path, e.g.: absolute: /usr/www/htdocs/yourdomain/cgi-bin/admin relative: admin 6. "Name of log file:" Define the log file name. If the program detects a predefined visitor/spider and blocks them from access, a log file entry is generated in the file specified in this field. Take note that you may only define the file name proper (e.g. "logs.txt"), *NOT* the file's directory path! 7. "Delimiter:" The delimiter is constituted by a character or a string placed inbetween single fields of a log file entry. (Default delimiter is: " -- ".) You may chose any character you prefer, however the delimiter should not appear in the logged data itself! Commonly used delimiter are: "|", "^", "§", etc. The configurable delimiter facilitates import of the log file into your local database (Access, Excel, etc.) for further processing and analysis. 8. "Name of host list file:" Define the name of the file containing the Hosts you wish to block. Take note that you may only define the file name proper (e.g. "hostlist.txt"), *NOT* the file's directory path! 9. "Name of IP list file:" Define the name of the file containing the IPs you wish to block. Take note that you may only define the file name proper (e.g. "iplist.txt"), *NOT* the file's directory path! 10. "Name of UserAgent list file:" Define the name of the file containing the UserAgents you wish to block. Take note that you may only define the file name proper (e.g. "ualist.txt"), *NOT* the file's directory path! 11. "Name of generated CGI Script:" Define the name of the CGI Script you wish to generate. (Default name is: "mb.cgi"). 12. "Temporary script directory:" After generation, the script must be saved. This will frequently cause problems related to system specific directory and file permissions. Let's assume you want to put the script "mb.cgi" in your main or root directory to call it from there. Typically, permissions for this directory will be restricted in such a way that only authorized users may upload a file there, e.g. via FTP. This is considered good security practice and hence should not be waived. Hence, the program (not holding "root" rights under Unix, "administrator" rights under NT) would be not be permitted to save the script in this directory. As a workaround you can create this temporary subdirectory and assign it "world writable" (Unix: "chmod 777") or "Write" (NT) permissions in order to save the script temporarily there. From here, you can then copy the script prior to initial usage manually (via Telnet or FTP) into its final directory. Before activating it, please don't forget to assign the correct file permissions (see above)! Alternative Option ------------------ Alternatively, instead of specifying a temporary script directory, you may enter the name of the final script directory in this field (see Step #13 below), provided this directory holds permissions "chmod 777" (Unix) or "Write" (NT). This would save you having to download the file to your local system and upload it again via FTP or, under Telnet, having to log into your shell and moving the script file to its target directory. Caution: Please take note that in this case you may inadvertently overwrite a pre-existent version of the script. Hence, take care to chose a new script name before generating the script! In any case BOTH fields ("Temporary script directory:" AND "Final script directory:") must be filled out as the program will otherwise generate an error message. 13. "Final script directory:" Specify the directory from which will actually run the generated script later. (Please study the explanations in Step #12 above if this should be unclear to you.) This entry is required to generate the correct HTML/SSI code. E.g. if you enter "cgi-bin" in this field, the following HTML/SSI code will be generated: In this example, a RELATIVE path is specified. Hence, the Final Script Directory must be located BELOW the HTML page in order to be able to call the blocker script! However, you may also define an ABSOLUTE path instead, e.g. "/cgi-bin". The slash "/" symbolizes the path's root, i.e. your domain's main directory. Hence, if you wish to install the script in your domain's main directory, simply enter "/" (without the quotes) in this field. 14. "Sleep time in seconds:" Here, you can define the duration of the time span you wish to put your targeted visitors to "sleep". During this phase, the server will postpone serving the HTML page content to the visitor. Experience has shown that a sleep phase of 3600 seconds (1 hour) will generally suffice - this is long enough to send the blocked visitor packing! Should the targeted visitor (human or robot) attempt to access your site again, they will simply be "put to sleep" once more for the time span specified. ------------------------------------------------- Email Alert (currently supported under Unix only) ------------------------------------------------- 15. "Send Email Alert:" (currently Unix only) Checking this box will activate the program's Email Alert function. When activated, you will receive a detailed email notification immediately whenever a blocked visitor should attempt to access your system. The following fields (Steps) #16-20 are email related and need only be filled in if you have chosen to activate the Email Alert function. If you have not activated this function, you may skip these steps and proceed with Step #21 below. 16. "Path to Sendmail:" (currently Unix only) Enter your system's path to Sendmail which will be used to administer the alert function. 17. "Subject for Email Alert:" (currently Unix only) Define the email alert message's subject line here. 18. "Email Alert from:" (currently Unix only) Specify the email alert sender's address in this field. This must be a syntactically valid email address, else the program will generate an error message. 19. "Email Alert to:" (currently Unix only) Specify where you want the email alert message to be sent. This must be a syntactically valid email address, else the program will generate an error message. 20. "CC:" (currently Unix only) If you wish to send a "carbon copy" (cc:) of the email alert to another address, you may enter it here. This must be a syntactically valid email address, else the program will generate an error message. -------- Email Alert Parameters End Here -------- 21. "Message to blocked Spiders (max. 256 chars.):" Optionally you can specify a message to send to a blocked visitor/spider here. Simply enter the text here (max. 256 characters). ----------------- Lists Maintenance ----------------- In the next section, you can edit the Hosts, IPs and UserAgents you want to block. In order to enable you to edit the lists for differently configured scripts, this editing function can be performed independently of selected blockage options. Hence, you can edit and save the lists without selecting one or more blockage functions. 21. "Hosts List" Enter the Hosts or Host ranges you wish to block here, one entry per line. E.g: cr000.digital-integrity.com cr001.digital-integrity.com .ro 22. "IP List" Enter the IPs or IP ranges you wish to block here, one entry per line. E.g: 62.200.157.32 198.124.64.17 198.138.63. 23. "UA List" Enter the UserAgents or UserAgent ranges you wish to block here, one entry per line. E.g: EmailCollector EmailSiphon Zeus --- Note ---- It is permissible to enter blank lines or lines starting with the comment hash # - this enables you to easily structure even very long lists. --- You may save the lists individually by clicking the buttons "Save Hosts", "Save IPs" or "Save UserAgents" respectively. Alternatively, you can save all lists in one go by clicking "Save ALL LISTS". Note that if you delete the content of a list in the text area fields completely and click on "Save" you will effectively delete the file's content. This will leave you with a blank lists file. --- Next, all entries will be save to a file as defined under variable "$parms_file" in the multiblocker-e.cgi script. (Default is: "varparm_mb.txt"). The current entries will always be saved in total when clicking one of the following buttons: "Save Hosts" "Save IPs" "Save UserAgents" "Save ALL LISTS" or "Generate Script" --- After filling out all required fields, click button "Generate Script". You can check the result in the text area field "Generated script code:". In the bottom text area field "HTML header code:" the generated HTML/SSI Code will be displayed. This code must be implemented in the HTML pages you wish to protect. We recommend cutting and pasting this code right at the top of your HTML pages, preferably above the first
tag. The reason is that when a visitor/spider hits your page, only the content part above the HTML header code will be displayed. ====================================================================== ERROR HANDLING -------------- This section covers individual error messages. Selection of option ------------------- "Please select blockage option!" Email adresses -------------- "Sender Email address ... is not valid!" "Target Email address ... is not valid!" "cc: Email address ... is not valid!" Email address will be checked for permissible characters. The following syntax is mandatory: Before @ character: letters, numbers, "_", "-", "+", "." After @ character in subdomains: letters, numbers, "_", "-" in top level domains (at the end) 2-4 letters Create admin directory ---------------------- "Please create directory: ...!" Open parms file --------------- "Variable parms file ... could not be opened! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Save parms file --------------- "Parms file ... could not be created! Please check permissions of ... directory: UNIX: "chmod 777" i.e. [drwxrwxrwx] NT: Deactivate "Read-only" status." "Cannot write to variable parms file ...! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Open host list file ------------------- "Host list file ... could not be opened! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Save host list file ------------------- "Cannot write to host list file ...! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Open IP list file ----------------- "IP list file ... could not be opened! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Save IP list file ----------------- "Cannot write to IP list file ...! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Open UserAgent list file ------------------------ "UserAgent list file ... could not be opened! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Save UserAgent list file ------------------------ "Cannot write to UserAgent list file ...! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." Temporary script directory -------------------------- "Please create temporary script directory: ...!" Saving the target file ---------------------- "Script file ... could not be created! Please check permissions of ... directory: UNIX: "chmod 777" i.e. [drwxrwxrwx] NT: Deactivate "Read-only" status." "Cannot save script file ... ! Please check file permissions: UNIX: "chmod 666" i.e. [-rw-rw-rw-] NT: Deactivate "Read-only" status." ====================================================================== KNOWN ISSUES ------------ Graphics -------- Graphics files uploaded to the CGI directory or to a directory below same may not be displayed correctly under some web server configurations. In this case you may create a directory outside of the cgi-bin. You can then define the "$graphics_dir" variable in program file multiblocker-e.cgi accordingly. Example: $graphics_dir = "../graphics/"; Docs (Manual/Help files) ------------------------ If the help file is not displayed correctly, we recommend uploading it to an alternate directory (outside of cgi-bin!) as well. You can then define the "$doc_dir" variable in program file multiblocker-e.cgi accordingly. Example: $doc_dir = "../docs/"; ====================================================================== UPDATES + PROGRAM CHRONOLOGY ============================ 2001-07-14: official release of version 1.01.01-e ====================================================================== CONTACT + SUPPORT ================= Please send email to: techsupport@fantomaster.com Corporate contact info: For an overview of our business hours in relation to your specific time zone, please see: < http://fantomaster.com/index2.html#hours > ###################################################################### # (c) Copyright 2001 by fantomaster.com # # All rights reserved. # # Copying, modification or distribution requires permission # # in writing by copyright holder. # # fantomas multiBlocker(TM) is the protected trade mark of # # fantomaster.com GmbH. # # URL: # # ------------------------------------------------------------------ # # OEM Licensing: # # ------------------------------------------------------------------ # # Affiliate Partner Program: # ######################################################################