Synchronising Categories and Directories

From EIRMA Wiki

Quick Links
Overview and Tutorial Introduction Quick Tutorial
Pre-Installation Site Configuration Installation Installing LDM Upgrading Removing Post-Installation Speed, Security and Server Load Uploads, Downloads and Media Filetypes
Administration Introduction Control Settings Permissions Controlling Access Monitoring and Controlling Usage Database Imports and Exports Content Overview/Mass Editing Entry Attributes Where are Files Stored?
Moderation Introduction Design Layout and Design Advanced vbAdvanced Subscriptions Add-on Extras Media Players Mirrors Synchronising Categories and Directories Hacking LDM Search Engine Optimisation
Other Known Bugs Common Problems Wish List Version History Acknowledgements

Contents 1 Example 2 In Detail 3 Notes

LDM can set up category structures that match the directory/file structure on your server, and keep the entries in these categories in sync with the contents of the web server directories. This can be done in two ways:

• Manually, using the scan command within LDM->Admin->Mass Edits. This requires that you first set up a desired category structure.
• Automatically, using the Synchronise with field on the Add/Edit Category form and the Controls on Category Synchronisation settings on LDM->Admin->Settings page or on Add/Edit Category->Control Settings. In this case, LDM can also build the category structure automatically.

Synchronisation is restricted to directories that are accessible based on the LDM security settings local_file_root and local_file_root_prefix (see Server Security) and by php's open_basedir. For example, if you instruct LDM to synchronise with a category called Music when local_file_root_prefix is set to /dir1/dir2/dir3, LDM will look for files in /dir1/dir2/dir3/Music.

Categories set up to synchronise do not offer the normal "Add Entry" command. In other words, a category can either accept entries manually or can synchronise, but not both at the same time.

Example

Suppose you create a category My Files and set it up to synchronise automatically with directory /myfiles using the following control settings:

sync_enabled : y
sync_freq : 10800
sync_filetypes : pdf, mp3
sync_populate_categories : y
sync_populate_entries : y
sync_maxcpu_allowed : 3
sync_maxcpu_allowed_manual : 45
sync_depth : 4

With these settings:

• LDM will re-synchronise (settings sync_populate_entries and sync_filetypes) pdf and mp3 files in /myfiles with My Files (provided pdf and mp3 files are defined in the vBulletin attachments table)
• The category will be resynchronised at most every 3 hours (10800 seconds, but perhaps much less often if no-one visits the category or its parents)

In Detail

Synchronisation is a potentially time-consuming process, so LDM limits the amount of time it spends on the process. Each time it displays a synchronised category:

• It checks the last time synchronisation was completed. LDM will only attempt a resync if this was more than sync_freq seconds ago. It will complete as much of this task as is possible in sync_maxcpu_allowed seconds.
• If sync_populate_categories) is set, LDM will also create and synchronise subcategories, scanning subdirectories to a depth of sync_depth and naming the subcategories according to the subdirectories. (Since subcategories inherit their parent's setting by default, if the user visits one of these subcategories when synchronisation is due, LDM will also scan sync_depth levels. Eventually, the process will have built up a complete directory listing.)
• If LDM fails to complete the process to the required depth, it will be repeated the next time someone views the category. (Consequently, for directories containing large numbers of files and subdirectories, users only see some of what is available until there have been sufficient visits to complete the synchronisation.)

Administrators also see a Manual Resynchronisation link in the category header, which they can use to force a resynchronisation at any time. Manual resynchronisations are allowed to use up to sync_maxcpu_allowed_manual seconds.

Notes

• Be careful: Synchronisation can have potentially expensive and unintended side effects when used indiscriminately on large directory trees
• Specify directories using local names (/files/music), not full urls. It is not possible to synchronise with files and directories held on another server
• When specifying directory names, keep in mind that the Admin->Settings local_file_root and local_file_root_prefix apply, see Server Security
• LDM will only locate files that have types listed in the vBulletin Attachments/Filetypes table, see Configuring Filetypes in vBulletin. The LDM->Admin->Setting/Edit->Category sync_filetypes limits, not extends, these filetypes.
• For security reasons, when setting up an automatic synchronisation, the user is not informed about the available server directories, whereas (since these pages are only available to administrators), manual synchronisations provide a drop-down menu that enable administrators to walk through the directory tree to select the files they wish to add to LDM.

• When categories are set to synchronise automatically, they will not accept manual entries. If you discontinue synchronisation by clearing the Synchronise with field, the category will keep its current enties and accept new manual entries again.
• Going the other way (changing a category that already contains entries to one that should synchronise) will delete all entries in the category except those that point to files in the synchronisation directory.