The SKIRT project
advanced radiative transfer for astrophysics
Public Member Functions | List of all members
StoredTable< N > Class Template Reference

#include <StoredTable.hpp>

Public Member Functions

 StoredTable ()
 
 StoredTable (const SimulationItem *item, string filename, string axes, string quantity, bool clampFirstAxis=true, bool resource=true)
 
 ~StoredTable ()
 
template<typename... Values, typename = std::enable_if_t<CompileTimeUtils::isFloatArgList<N - 1, Values...>()>>
double cdf (Array &xv, Array &pv, Array &Pv, Range xrange, Values... values) const
 
void open (const SimulationItem *item, string filename, string axes, string quantity, bool clampFirstAxis=true, bool resource=true)
 
template<typename... Values, typename = std::enable_if_t<CompileTimeUtils::isFloatArgList<N, Values...>()>>
double operator() (Values... values) const
 
template<typename Value , typename = std::enable_if_t<N == 1 && CompileTimeUtils::isFloatArgList<1, Value>()>>
double operator[] (Value value) const
 
template<typename... Values, typename = std::enable_if_t<CompileTimeUtils::isFloatArgList<N - 1, Values...>()>>
void valueArray (Array &yv, const Array &xv, Values... values) const
 

Detailed Description

template<size_t N>
class StoredTable< N >

An instance of the StoredTable<N> class template provides access to the contents of a particular resource file in the SKIRT stored table format (i.e. a "stored table").

Stored table file format

A stored table includes the names of the quantities on the axes (e.g. wavelength and grain size) and those being tabulated (e.g. absorption and scattering efficiencies), in addition to the grid points for each axis and the tabulated data values. All grid points and values are stored as binary data in the form of 64-bit floating-point numbers, and are always given in SI units. The format is designed so that it is easy to calculate the offset, relative to the start of the file, to any particular data value. More specifically, a stored table file is essentially a sequence of 8-byte data items. A data item can have one of three types:

The overall layout is as follows:

The values are ordered so that the quantity values for a particular point are next to each other, the first axis index varies most rapidly, and the last axis index varies least rapidly.

The StoredTable<N> class template

The StoredTable<N> template parameter N specifies the number of axes in the stored table, and thus the number of axis values needed to retrieve a tabulated quantity from the table. Each StoredTable<N> instance represents a single tabulated quantity. Accessing the full contents of a stored table resource file with multiple tabulated quantities requires a seperate StoredTable<N> instance for each of those quantities.

The default constructor creates an invalid stored table instance. The alternate constructor and the open() function associate a particular stored table resource file with the stored table instance. The number of axes in this stored table resource file must match the template parameter N. Also, the axis names and the corresponding units in the file, and one of the tabulated quantity names and its corresponding unit in the file, must match the information passed to the alternate constructor or the open() function. The destructor automatically releases the file association and any related resources.

The parenthesis operator returns the quantity represented by the StoredTable<N> for the N specified axis values, interpolated from the tabulated values. Other functions offer access for specific purposes, such as constructing a cumulative distribution function along one axis, given values for the other axes.

Implementation and performance

A StoredTable<N> instance acquires a read-only memory map on the associated stored table resource file as opposed to actually reading the file contents into memory through regular file I/O operations. This has some important, mostly positive, consequences.

Acquiring a memory map establishes a mapping between "pages" of system-defined size in the logical address space of a process and the contents of the "backing file", in this case the stored table resource file. This operation is simple and thus very fast. From then on, the operating system automatically loads pages from the backing file into physical memory as they become needed because the program addresses an item in the logical memory range of the page. Conversely, the operating system automatically removes pages from physical memory if available memory becomes tight. In effect, the operating system automatically manages a high-performance caching mechanism on stored tables.

Three important use cases benefit greatly from this mechanism. Firstly, a large resource file can be left associated with a StoredTable<N> instance for the duration of the program, even if it is used only sporadically. When memory is tight, infrequently used portions of the data will automatically be removed from memory and reloaded later if needed. Secondly, there is little overhead in constructing a StoredTable<N> instance (and possibly destroying it shortly thereafter) even when the program needs only a small portion of the file contents. And thirdly, because all StoredTable<N> instances associated with a given stored table resource file share the same memory map on that file, using a seperate instance for each quantity in the stored table incurs very little overhead.

Moreover, most operating systems share memory maps between processes. For a parallel program using MPI, this means that all processes running on the same compute node share a single memory copy of the resources they employ. Also, most operating systems keep the memory map caches alive between consecutive invocations of a program (assuming memory is available), increasing performance when, for example, interactively testing the program.

On the downside, a program requesting a huge chunk of data from a large stored table in a serial fashion would run faster using regular file I/O, because the separate page loads take more time than sequentially reading data in bulk. More importantly, performance usually degrades rapidly (to the point where the program no longer performs any useful work) when the system is forced to constantly remove and reload pages because there is not enough memory to hold the data needed for a particular phase in the program. And finally, the run-time performance of a program becomes somewhat unpredicable because the speed of accessing resources depends heavily on the previous state of the operating system caches.

Constructor & Destructor Documentation

◆ StoredTable() [1/2]

template<size_t N>
StoredTable< N >::StoredTable ( )
inline

The default constructor constructs an invalid stored table instance. The user must call the open() function to associate the stored table instance with a particular stored table resource file. Calling any of the other functions before calling open() results in undefined behavior (usually a crash).

◆ StoredTable() [2/2]

template<size_t N>
StoredTable< N >::StoredTable ( const SimulationItem item,
string  filename,
string  axes,
string  quantity,
bool  clampFirstAxis = true,
bool  resource = true 
)
inline

This alternate constructor constructs a stored table instance and immediately associates a given stored table resource file with it by calling the open() function. Refer to the open() function for a description of the arguments and of its operation.

◆ ~StoredTable()

template<size_t N>
StoredTable< N >::~StoredTable ( )
inline

The destructor breaks the association with a stored table resource file established by the alternate constructor or the open() function, if there is any. In practice, this simply means releasing the memory map on the associated file.

Member Function Documentation

◆ cdf()

template<size_t N>
template<typename... Values, typename = std::enable_if_t<CompileTimeUtils::isFloatArgList<N - 1, Values...>()>>
double StoredTable< N >::cdf ( Array xv,
Array pv,
Array Pv,
Range  xrange,
Values...  values 
) const
inline

This function constructs both the normalized probability density function (pdf) and the corresponding normalized cumulative distribution function (cdf) for the tabulated quantity across a given range in the first axis (the xrange argument), using given fixed values for the other axes, if any (the arguments at the end of the list).

The resulting first-axis grid is constructed into xv, the corresponding pdf into pv, and the corresponding cdf into Pv. In all cases, xv[0]=xmin, xv[n]=xmax, Pv[0]=0, and Pv[n]=1. The function returns the normalization factor, i.e. the value of Pv[n] before normalization.

If any of the axes values, including the xrange values specifying the range for the first axis, are out of range of the internal grid, extra quantity values are fabricated according to the policy set by the clampFirstAxis flag in the open() function.

If the flags specified in the stored table indicate that both the first axis and the quantity represented by the table should be interpolated logarithmically, it is assumed that the pdf behaves as a power-law between any two grid points, and the integration to determine the cdf is performed accordingly. In all other cases, piece-wise linear behavior is assumed and regular trapezium-rule integration is used.

◆ open()

template<size_t N>
void StoredTable< N >::open ( const SimulationItem item,
string  filename,
string  axes,
string  quantity,
bool  clampFirstAxis = true,
bool  resource = true 
)
inline

This function associates a given stored table resource or input file with the stored table instance. If such an association already exists, this function throws a fatal error. Conversely, calling any of the other functions before an association exists results in undefined behavior (usually a crash).

The item argument specifies a simulation item in the hierarchy of the caller (usually the caller itself) used to retrieve an appropriate logger.

If the resource flag is true (the default value), the filename argument specifies the filename of the built-in resource, without any directory segments. If the resource flag is false, the filename argument specifies the file path of the stored table input file relative to the input path of the simulation. In both cases, the file must have the ".stab" filename extension, which will be added to the specified filename if needed.

The filename argument specifies the filename of the resource, without any directory segments. The resource file must have the ".stab" filename extension, which will be added to the specified filename if needed.

First of all, the number of axes in this stored table resource file must match the template parameter N. Furthermore, the axes names in the resource file and the corresponding units for each axis must match the information specified in the axes argument. Finally, one of the tabulated quantity names in the resource file and its corresponding unit must match the information specified in the quantity argument. For a stored table resource file with multiple tabulated quantities, the quantity argument at the same time determines which of these quantities will be associated with the stored table instance.

The string passed to the axes argument must have the syntax "name1(unit1),...,nameN(unitN)". In other words, a unit string between parenthesis follows each axis name, and the specifications for different axes are separated by a comma. For example, "lambda(m),a(m)". Whitespace is not allowed. The string passed to the quantity argument must have a similar syntax, for a single name/unit combination. Examples include "Llambda(W/m)", "Qabs(1)", and "h(J/m3)".

By default, out-of-range axes values are clamped to the corresponding outer grid point. In other words, a "nearby" quantity value at some outer grid point is used for out-of-range axes values. This behavior can be turned off for the first axis (but not for the other axes) by passing a value of false to the optional clampFirstAxis flag. In that case, the quantity value for out-of-range first-axis values is considered to be zero.

In summary, this function (1) locates the specified stored table resource file, (2) acquires a memory map on the file, (3) verifies that the stored table matches all requirements, and (4) stores relevant information in data members. If any of these steps fail, the function throws a fatal error.

◆ operator()()

template<size_t N>
template<typename... Values, typename = std::enable_if_t<CompileTimeUtils::isFloatArgList<N, Values...>()>>
double StoredTable< N >::operator() ( Values...  values) const
inline

This function returns the value of the quantity represented by this stored table for the specified axes values, interpolated over the grid points of the actual tabulated values in all dimensions. The function uses linear or logarithmic interpolation for the axes and quantity values according to the flags specified in the stored table. Out-of-range axes values are handled according to the policy set by the clampFirstAxis flag in the open() function.

◆ operator[]()

template<size_t N>
template<typename Value , typename = std::enable_if_t<N == 1 && CompileTimeUtils::isFloatArgList<1, Value>()>>
double StoredTable< N >::operator[] ( Value  value) const
inline

For a one-dimensional table only, this function returns the value of the quantity represented by the stored table for the specified axes value, interpolated over the grid points of the actual tabulated values. The function uses linear or logarithmic interpolation for the axis and quantity values according to the flags specified in the stored table. Out-of-range axes values are handled according to the policy set by the clampFirstAxis flag in the open() function.

◆ valueArray()

template<size_t N>
template<typename... Values, typename = std::enable_if_t<CompileTimeUtils::isFloatArgList<N - 1, Values...>()>>
void StoredTable< N >::valueArray ( Array yv,
const Array xv,
Values...  values 
) const
inline

This function returns a sequence of values of the quantity represented by this stored table (in the yv argument) corresponding to the specified sequence of first-axis values (the xv argument), using given fixed values for the other axes, if any (the arguments at the end of the list). The function behaves as if it would call the operator()() function for each element in the specified input sequence, but it is more efficient.


The documentation for this class was generated from the following file: