edu.unh.sdb.datasource

Class SparseDataSource

java.lang.Object

edu.unh.sdb.datasource.DataCollection

edu.unh.sdb.datasource.DataSource

edu.unh.sdb.datasource.PhysicalDataSource

edu.unh.sdb.datasource.BinaryFileDataSource

edu.unh.sdb.datasource.SparseDataSource

All Implemented Interfaces:

Indexable

public class SparseDataSource
extends BinaryFileDataSource

Implements a data source that only contains data for a subset of its bounds. The parent class partitions the bounds space into fixed size blocks except for the border blocks that can be smaller if the block size does not divide exactly into the data source bounds. This version maintains a cache of blocks and an LRU replacement algorithm. The default # partitions is 64. IMPORTANT: the physical blocks in the sparse data source are all the same size, so that the location of any physical block can easily be converted to a seek address in the file. This means that partial blocks will be null padded in the physical file. The bounds of the SparseDataSource define the bounds of the true data, so the code can easily figure out what blocks are partial; for the time being, the mapper data puts the size of all non- standard blocks in the xfdl file. This is intended for use with the simulation interface to AR data, but could become a standard AR tool. The xfdl file for a sparse data source includes a mapping of blocks in the virtual DataSource to blocks in the physical file. For now, access to non-existent locations in the virtual DataSource geometry are considered to be an error. Later, we can add some methods that allow program to specify that missing data is to be replaced by zero or some other value -- probably attribute specific.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SparseDataSource.MapInfo

Field Summary

Fields

Modifier and Type	Field and Description
protected ISBounds	blockBounds
protected java.util.HashMap<java.lang.Integer,java.lang.Integer>	blockMapper
protected RegularISPartitioning	blockPartitioning
protected IndexSpaceID	blockSizes
protected long	cacheRefTime
protected java.util.HashMap<java.lang.Integer,edu.unh.sdb.datasource.SparseDataSource.DBCache>	dbCache
protected SparseDataSource.MapInfo[]	mapInfo
protected java.lang.String[]	mappingStrings
protected int	maxCacheSize
protected int	maxCacheSizeDefault
protected DataSource	origDS
protected int	partitionByteSize
protected ISBounds	partitioningBounds

Fields inherited from class edu.unh.sdb.datasource.BinaryFileDataSource

bufferIterCopy, bufferUpper, dimensions, fChannel, iniOffset, iterCopy, leastSignificantAxis, raFile, recordByteSize, recordNumFields, srcRodByteSize, srcRodSize, steps, tempDatum

Fields inherited from class edu.unh.sdb.datasource.PhysicalDataSource

BIG_ENDIAN, bufferBounds, bufferDimension, bufferSteps, byteBuffers, directMemoryLimit, endian, LITTLE_ENDIAN, maxBufferSize, maxGapSize, readCounter, readLength, url

Fields inherited from class edu.unh.sdb.datasource.DataSource

fdlDescriptor, isActivated, name, wsName

Fields inherited from class edu.unh.sdb.datasource.DataCollection

attributeOrder, axisOrdering, axisWeights, bounds, dataOrder, dimensionality, numAttributes, pointOrder, recordDesc

Constructor Summary

Constructors

Constructor and Description
SparseDataSource(java.lang.String name, FDLDescriptor fdl) This is the mechanism for creating a sparse ds from an xfdl file, but it is not normally used by an app, which should use Datasource.create, which calls this method.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	addMap(int original, int sparse) addMap - add another block map entry
protected void	buildMapper(java.lang.String[] mapStrings) generate the block mapping hash table
void	datum(Datum dtm, IndexSpaceID did) Given an IndexSpaceID and FieldIDMapper, copy the values of the Datum at that position to the argument Datum.
void	datum(Datum dtm, IndexSpaceID did, FieldIDMapper mp) Given an IndexSpaceID and FieldIDMapper, copy the values of the Datum at that position to the argument Datum.
int	getBlock(IndexSpaceID loc) Return the block index (id) for the partition that contains the "loc" IndexSpaceID.
int	getMap(int original) getMap - get the sparse block id for the specified original block
ISBounds	getPart(IndexSpaceID loc) returns an array of ISBounds representing the pieces of the input bounds that have data in this sparse data source.
java.util.ArrayList<ISBounds>	getParts(ISBounds bnds) returns an array of ISBounds representing the pieces of the input bounds that have data in this sparse data source.
Values	getValues() Return a Values array of all attributes in whatever type and order they are stored internally (either attribute order or point order for Granite-defined objects).
protected void	initialize()
protected void	loadPartition(int partitionIndex, Values data) Given a partition index, read the data from the partition into a Values object.
static void	main(java.lang.String[] args)
protected void	setBlockBounds(ISBounds blkBounds) set/reset the size of the blocks; only do this if the physical file changes correspondingly -- not a public method.
void	setCacheSize(int nPartitions) Define the number of partitions to be kept in the cache memory.

Methods inherited from class edu.unh.sdb.datasource.BinaryFileDataSource

activate, deactivate, equals, getFileOffset, hashCode, setAxisOrdering, setDatum, setFileOffset, toString

Methods inherited from class edu.unh.sdb.datasource.PhysicalDataSource

allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, deleteBuffer, deleteBuffer, deleteBuffers, getBuffer, getBuffer, getBufferSize, getBufferSize, getByteOrder, getDataFileString, getMaxBufferSize, getReadCounter, getReadLength, getURL, initializeBuffers, isBigEndian, makeBestBuffer, resetReadCounter, setBufferByteOrder, setByteOrder, setMaxBufferSize, setMaxGapSize, subblock, subblock, subblock, toString

Methods inherited from class edu.unh.sdb.datasource.DataSource

checkActivated, clearCache, copyBestID, create, create, create, createRemote, createRemote, createRemoteClient, createRemoteClient, createRemoteDataSource, createRemoteDataSource, createRemoteDataSourceUDP, createRemoteUDP, createRemoteUDP, dumpData, getFDLDescriptor, getMetadataCollection, getName, getStatistics, isActivated, setMetadataCollection

Methods inherited from class edu.unh.sdb.datasource.DataCollection

byteSize, computeOffset, computeOffset, copyBounds, createDataBlock, createDataBlock, createDataBlock, createDatum, createDatum, createDatum, createDatum, createDatum, createDatum, datum, datum, datum, datum, datum, dim, getAttributeNames, getAxisOrdering, getAxisWeight, getAxisWeights, getBounds, getByte, getBytes, getBytes, getBytes, getBytes, getBytes, getBytes, getBytesByAttribute, getBytesByAttribute, getBytesByPoint, getBytesByPoint, getDataOrder, getDimensionsArray, getDouble, getDoubles, getDoubles, getDoubles, getDoubles, getDoubles, getDoubles, getDoublesByAttribute, getDoublesByAttribute, getDoublesByPoint, getDoublesByPoint, getFloat, getFloats, getFloats, getFloats, getFloats, getFloats, getFloats, getFloatsByAttribute, getFloatsByAttribute, getFloatsByPoint, getFloatsByPoint, getInt, getInts, getInts, getInts, getInts, getInts, getInts, getIntsByAttribute, getIntsByAttribute, getIntsByPoint, getIntsByPoint, getLowerBoundsArray, getNumAttributes, getRecordDescriptor, getShort, getShorts, getShorts, getShorts, getShorts, getShorts, getShorts, getShortsByAttribute, getShortsByAttribute, getShortsByPoint, getShortsByPoint, getStorageType, getUpperBoundsArray, getValues, getValuesByAttribute, getValuesByPoint, longVolume, setBounds, setRecordDescriptor, size, subblock, subblock, subblock, subblock, subblock, subblock, subblock, subblock, subblock, useBytesOnly, volume

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

blockBounds

protected ISBounds blockBounds

blockMapper

protected java.util.HashMap<java.lang.Integer,java.lang.Integer> blockMapper

origDS

protected DataSource origDS

mappingStrings

protected java.lang.String[] mappingStrings

mapInfo

protected SparseDataSource.MapInfo[] mapInfo

partitioningBounds

protected ISBounds partitioningBounds

blockPartitioning

protected RegularISPartitioning blockPartitioning

blockSizes

protected IndexSpaceID blockSizes

partitionByteSize

protected int partitionByteSize

dbCache

protected java.util.HashMap<java.lang.Integer,edu.unh.sdb.datasource.SparseDataSource.DBCache> dbCache

maxCacheSize

protected int maxCacheSize

maxCacheSizeDefault

protected int maxCacheSizeDefault

cacheRefTime

protected long cacheRefTime

Constructor Detail

SparseDataSource

public SparseDataSource(java.lang.String name,
                FDLDescriptor fdl)

This is the mechanism for creating a sparse ds from an xfdl file, but it is not normally used by an app, which should use Datasource.create, which calls this method. This is public unlike most equivalent Granite DataSource constructors -- this is probably a better model to provide an app with the opportunity to create a specific kind of DataSource dynamically -- compared to the complex and inconsistent alternative public constructors that we provide elsewhere.

Parameters:

name - - informal name for the ds

fdl - - fdl describing characteristics -- usually extracted from xfdl file.

Method Detail

setCacheSize

public void setCacheSize(int nPartitions)

Define the number of partitions to be kept in the cache memory.

Parameters:

nPartitions -

initialize

protected final void initialize()

setBlockBounds

protected void setBlockBounds(ISBounds blkBounds)

set/reset the size of the blocks; only do this if the physical file changes correspondingly -- not a public method.

buildMapper

protected void buildMapper(java.lang.String[] mapStrings)

generate the block mapping hash table

addMap

protected void addMap(int original,
          int sparse)

addMap - add another block map entry

getMap

public int getMap(int original)

getMap - get the sparse block id for the specified original block

getBlock

public int getBlock(IndexSpaceID loc)

Return the block index (id) for the partition that contains the "loc" IndexSpaceID.

Parameters:

loc -

Returns:

getParts

public java.util.ArrayList<ISBounds> getParts(ISBounds bnds)

returns an array of ISBounds representing the pieces of the input bounds that have data in this sparse data source.

getPart

public ISBounds getPart(IndexSpaceID loc)

returns an array of ISBounds representing the pieces of the input bounds that have data in this sparse data source.

datum

public void datum(Datum dtm,
         IndexSpaceID did,
         FieldIDMapper mp)

Given an IndexSpaceID and FieldIDMapper, copy the values of the Datum at that position to the argument Datum. TODO Define error conditions

Specified by:

datum in interface Indexable

Specified by:

datum in class DataCollection

datum

public void datum(Datum dtm,
         IndexSpaceID did)

Given an IndexSpaceID and FieldIDMapper, copy the values of the Datum at that position to the argument Datum. Don't need to explicitly call super, but I'm doing that because I think we'll want to have our own code here.

Specified by:

datum in interface Indexable

Overrides:

datum in class DataCollection

getValues

public Values getValues()

Return a Values array of all attributes in whatever type and order they are stored internally (either attribute order or point order for Granite-defined objects). The default implementation is often overridden for efficiency.

Overrides:

getValues in class DataSource

loadPartition

protected void loadPartition(int partitionIndex,
                 Values data)

Given a partition index, read the data from the partition into a Values object. This code has been copied from ChunkedDataSource.loadChunk Since Partitions are likely to be much larger than chunks often are, I've used int instead of long for the partition index value. Also, I took out the loop that was in loadChunk just in case the buffer was not big enough -- since we are always reading exactly one partition, the buffer size should always be ok.

main

public static void main(java.lang.String[] args)