DUG Insight User ManualDUG Insight DUG Insight User ManualIT / Data Manager GuideHow DUG access data: random, concurrent, or STREAM (large data)

How DUG access data: random, concurrent, or STREAM (large data)

Insight stores project data differently based on the data type. For data which is frequently read or updated in small batches (the project database, for example), Insight will use single-file databases. For data which is seldom written but which must be searched and read in large blocks (seismic, for example), Insight will use an on-disk filesystem which is arranged to maximise the opportunity for large block reads.

In configuring Insight for best performance on a network, it's not unusual to store either type of file on a different filesystem. The typical method of achieving this is to first ensure that your project database and horizons are stored on your high performance concurrent filesystem. In the past we've seen good performance over NFS, and since these files are often a small fraction of project size, this can also be backed by SSD for optimal performance.

With those in place, the remaining files, which will primarily consist of volume data, are best served by a filesystem tuned for large reads. Where possible, data will be streamed out completely in order, but where not (such as when reading in a sort order not represented by the on-disk file), the file format will still serve blocks of data in order to reduce seek overhead.

You can configure the location of your project directories by editing the dug.project file found in each project directory. The directories section contains a number of entries allowing you to redirect project content. Typically, simply setting them to a subdirectory (like db and 120hors in the example below) is enough to then follow up with a filesystem configuration which redirects those paths.

directories
    statics statics
    userHorizons 120hors
    archive .
    importexport .
    wavelets wavX
    database db
    culture 191culture
    volumes 100sei
    base .
    horizons 120hors
    screenshots rep
Click to copy

Finally, Insight supports the option of launching a site-specific project configuration script whenever a project is created, in order for site-specific post-creation actions to occur, such as configuring filesystem support, or relocating and symlinking directories. To enable this functionality, set the INSIGHT_PROJECT_CREATION_HELPER environment variable to the path of the post-creation script before launching Insight or the DUG Launcher. Once in place, whenever a new project is created, the script will be run with the new project's location as its working directory.