Speed, Security and Server Load

From EIRMA Wiki

Jump to: navigation, search
Quick Links

Overview and Tutorial

Pre-Installation

Installation

Post-Installation

Administration

Moderation

Design

Advanced

Other

Contents


Introduction

LDM places little overhead on your server when used purely as a links manager. As installed, it is also secure and semi-leechproof. Users cannot easily discover where the downloads are stored unless you explicitly give them permission to edit entries or make them members of the administrator user group.

You should think about security and server load before offering controlled downloads of files that you really want to keep secure; before enabling uploads; and before constructing complex vBadvanced pages involving several LDM vBa modules. Consider these points:

  • Where will the files be stored on the server?
  • How important is it to keep file locations hidden (from your users and to prevent leeching by others)?
  • Is server load a serious issue?
  • How concerned are you about bandwidth?
  • Are you willing to lower the security settings in order to reduce the load on your server?
  • Do you want to allow your user community to upload files?
  • Are you confident that rogue users will not place viruses and other malicious items on your site?

Standard settings steer a middle course:

  • File locations are kept hidden from the end user.
  • The downloads hosted on your server and the files that users upload are held within the web site's own directory structure. This makes them - in principle but with difficulty - accessible to informed guesswork.
  • LDM retains full control of the download process by managing the transfer of files to the client.
  • Only administrators can add/edit entries to the database.
  • Display of thumbnail images is disabled.
  • Displayed material is regenerated on every page display.

You may need to change these settings in situations such as the following:

  • Before offering (allowing your users to offer) downloads of large files (multi-megabytes) that are actually held on another web site. In this situation, the default settings place a high cpu overhead on your server and double your site's use of bandwidth.
  • You want to prevent a malicious user from guessing where files are stored, so that he can bypass LDM altogether.
  • You want to use vBadvanced as your main portal and include several LDM vBa modules on this portal.
  • You include several LDM 'statistics' in the main LDM pages.
  • You have a complex category structure.

Ways of handling these situations are explained below.

Caching Display Bits

By default, LDM regenerates all pages each time they are displayed. There are a number of reasons for choosing this default, notably that it guarantees that all displays will correctly reflect complex user permissions and will always be up-to-date. However, this places a load on your server, which is most noticable when you use vBadvanced and have several LDM modules displayed on the page.

From version 2.3.0, LDM can cache a number of key displays, including the LDM vBadvanced modules, certain category listings and some other items. This feature is enabled and controlled using the bit_cache_active, bit_cache_expiry and bit_cache_usergroup_bypass LDM admin settings. Caching is carried out on a per-usergroup, per-style basis, i.e. when an item is cached, it is tagged with the current user's usergroupid and styleid, and will only be presented to other users with the same primary usergroup and style selection.

Caching can achieve a significant reduction in server load. For example, vBadvanced pages involving multiple LDM modules can be displayed ten times faster and with a fraction of the database accesses when the LDM modules have been cached. However, not all material is suitable for caching, and a balance has to be achieved between the cpu time required to generate a display and the memory overhead of maintaining a large cache. Further experiments are planned to explore the best balance.

It is possible, depending on your site's access permissions, that caching will display (or hide) entries and categories that would otherwise not be shown (be shown) to a given user. Although this does not make restricted entries/categories accessible to the user, you may not wish this visibility to happen. Consequently, caching can be disabled on a per-usergroup basis with the bit_cache_usergroup_bypass LDM admin setting.

Cached items are regenerated whenever:

  • the bit_cache_expiry time (in minutes) has elapsed
  • when significant changes are made to the contents of the LDM database, notably that categories and entries are added, modified, deleted, emptied or moved using the admin/mass edit facilities and the standard add link/add category/ etc pages.

Minimising Server Load during Downloads

Potentially, LDM's most server-intensive task is to manage file downloads. Some types of download are more expensive than others, and links and downloads offer different trade-offs between efficiency and security.

In general, processing a link is more efficient than processing a download, because LDM is not providing the intermediate file handling. However, this improvement is achieved at the cost of a loss of security and control: the location of the file being transferred becomes visible to the prying user and some filetypes may not transfer in the way you intend.

Recall that LDM defines downloads as transfers of files that have types defined in the vBulletin attachment table. Filetypes that are not defined in this table are handled as links. Since access to links is controlled by triggering browser redirects, with no specal management or attempt to keep the location secret, there is no server load involved.

For downloads, server load increases according to the size of the file. Entries that have been set up to point to files specified using full urls are inherently more expensive to transfer than files held on the local server and specified using just the local filename. (This is true even if the url actually points to the local site.)

You can minimise server load in several ways:

  • By keeping downloads on the local server and identifying them with a local filename (or, equivalently, using the upload facility to transfer files to LDM.)
  • By adjusting the settings described below to balance load and secrecy
  • By disabling server intensive features
  • By enabling the caching features

Balancing Server Load and Secrecy

You can effectively eliminate the server load associated with processing downloads using LDM->Admin->Settings force_redirect. Setting force_redirect to 1 or 2 reduces or eliminates the overhead involved in handling downloads, but (potentially or immediately) reveales file locations to the prying user.

  • force_redirect=0 (default setting) : This provides the tightest security. LDM reads the contents of files and retransmits the contents to the user in a controlled and fully secure fashion. For reasonably-sized local files, this approach does not place a significant load on the server. However, fetching the file from another site increases the server load substantially and doubles your bandwidth consumption.
  • force_redirect=1 (or the entry is a link rather than download) : LDM keeps the location of the file hidden on its own displays but delivers the content by telling the user's browser where to find the file. This approach (browser redirect) reveals the file's location to the curious and informed user. However, since the actual file delivery is not handled by LDM, the load on your server and the bandwidth required to transfer the file are kept to a minimum.
  • force_redirect=2. LDM displays direct html links to these files, so the location is obvious and visible to the user. In this situation, LDM plays no part in delivering the file to the user, so there is no overhead. Note that some LDM features, such as hit recording and use of mirror sites, are not available when this option is selected.

Local Files versus Fully-Qualified URLs

When force_redirect is set to 0, you should always use local filenames rather than fully-qualified urls for files that are stored on your server. Downloading fully-qualified urls will increase the load on your server significantly. In this situation, LDM is likely to consume its permitted cpu time (typically 30 seconds) for file larger than a few megabytes. Furthermore, since the server has to read the file from the remote site machine and retransmit it to the user, bandwidth consumption is doubled. This is true, even if the "remote site" is actually your own.

On a typical system, case 1) seems to run about 100 times faster than case 3), which requires roughly 1 second cpu to handle a 250kb remote file. These figures vary according to whether you are running Linux or Windows and the speed of the server cpu and the network.

Note that LDM itself cannot optimise the middle case by recognising that the file is actually on your own server and removing http://yoursite from http://yoursite/directory/file.typ, because your web server may be translating urls, for example using the Apache mod_rewrite module.

Disabling Worst Case Transfers

Turn off allow_remote_downloads to prevent use of LDM for accessing downloads held offsite, and thereby eliminate the worst-case situation for server load. Entry of, and access to, remote urls that point to downloadable files, i.e. files with recognised mime types, will be blocked. LDM will still accept urls that do not refer to a downloadable file, as well as files held on the server itself.

Increasing Server Security

By default, LDM expects that any local files that it manages will be stored in subdirectories within the website directory structure. You can increase security by storing these files outside the website directories. The following settings control which directories LDM uses to store and access local files:

local_file_root 
When set to 'yes', the location of local files is specified relative to the root of local file system rather than relative to the root of the website. This allows you to store files outside the website, making these files inaccessible to the prying user. Use of this setting requires careful attention to security to avoid opening up your server to malicious users.
local_file_root_prefix 
This defines a prefix (e.g. '/downloads') to be inserted at the start of each local file name. This is used to modify the base point relative to which files are stored and located. This limits LDM access to a particular part of the (web or local) directory tree, removing any possibility for users to create arbitrary links to other parts of your site or server.
secure_nullindexfile 
Enabling this setting causes LDM to create an empty index.html file in each upload directory. This can help prevent prying users from guessing where uploads are stored and then using the browser to scan the directory. (Note that there are better ways to achieve this result, such as disabling directory browsing altogether in the web server configuration file, or by using the local_file_root setting to place files outside the web server directory tree.

LDM applies local_file_root and local_file_root_prefix in combination. Consider an entry pointing to file /fred/joe.pdf

  • local_file_root is off and local_file_root_prefix is blank - LDM looks for a file http://yoursite/fred/joe.pdf
  • local_file_root is off and local_file_root_prefix is /jim - LDM looks for http://yoursite/jim/fred/joe.pdf
  • local_file_root is on and local_file_root_prefix is blank - LDM accesses /fred/joe.pdf in the top level directory on your web server. This potentially opens the entire file system to prying eyes.
  • local_file_root is on and local_file_root_prefix is /jim - LDM looks in server directory /jim/fred/joe.pdf

The local_file_root and local_file_root_prefix settings apply throughout LDM. For example, directory scans and category synchronisations can only search parts of the directory tree within the space defined by local_file_root and local_file_root_prefix.

Beware

  • Do not enable local_file_root unless you have full control over your web server. Never turn this setting on without also setting local_file_root_prefix, otherwise you open up your entire server to prying eyes.
  • If you change these settings after loading files into your LDM database, you will either need to edit entries to reflect the new settings or physically move the files so that the links remain valid. The "mass edit" functions in Administer->Categories provides a quick way to do this.
  • If the PHP open_basedir setting is defined, LDM's access to files on the local server is always restricted. In this case, you must ensure that the values of local_file_root and local_file_root_prefix are consistent with the open_basedir setting.

Preventing Leeching

Leeching occurs when another site provides unauthorised links to your downloads, using your server to host the files and provide the bandwidth. You can prevent it by enabling the LDM->Admin->Settings->Security Tags. This adds an encrypted tag to download urls, which defines who can access the download and when. Restrictions can be imposed by time (how long the url will be valid), by userid (who can access the file via this url) and by IP address (from where it can be accessed).

When this security feature is enabled, your user community can still copy and paste LDM urls that display entries in their categories (../local_links?catid=NN), but play and download urls (../local_links?action=play/jump) will be useless other than to the permitted users.

Blocking access to unregistered users

LDM's in-built access control mechanisms limit what your users can see, do and access. Howeever, if you want to completely block (e.g.) unregistered users from accessing LDM at all, create the following plugin, attached to the hook ldm_start 

if ($vbulletin->userinfo['usergroupid'] <= 1) {
  print_no_permission();
  exit;
}

Modify this code as required to block other usergroups. For example, this code allows access only to moderators and administrators: 

if (!in_array($vbulletin->userinfo['usergroupid'], array(5,6,7)) {
  print_no_permission();
  exit;
}
Retrieved from "http://www.eirma.org/wikis/index.php/Speed,_Security_and_Server_Load"
Personal tools