edu.unh.sdb.datasource

Class PhysicalDataSource

java.lang.Object

edu.unh.sdb.datasource.DataCollection

edu.unh.sdb.datasource.DataSource

edu.unh.sdb.datasource.PhysicalDataSource

All Implemented Interfaces:

Indexable

Direct Known Subclasses:

BinaryFileDataSource, ChunkedDataSource, FileDataSource, GraniteRemoteClient, RemoteBFDS, RemoteUDPClient, RemoteUDTClient, StatsDataSource, TextFileDataSource

public abstract class PhysicalDataSource
extends DataSource

Field Summary

Fields

Modifier and Type	Field and Description
static java.nio.ByteOrder	BIG_ENDIAN
protected ISBounds	bufferBounds
protected IntegerIndexSpaceID	bufferDimension
protected int[]	bufferSteps
protected java.nio.ByteBuffer[]	byteBuffers
static long	directMemoryLimit
protected java.nio.ByteOrder	endian
static java.nio.ByteOrder	LITTLE_ENDIAN
protected long	maxBufferSize
protected long	maxGapSize
protected long	readCounter
protected long	readLength
protected java.lang.String	url

Fields inherited from class edu.unh.sdb.datasource.DataSource

fdlDescriptor, isActivated, name, wsName

Fields inherited from class edu.unh.sdb.datasource.DataCollection

attributeOrder, axisOrdering, axisWeights, bounds, dataOrder, dimensionality, numAttributes, pointOrder, recordDesc

Constructor Summary

Constructors

Constructor and Description
PhysicalDataSource(java.lang.String name, FDLDescriptor fdl) Create a PhysicalDataSource from an fdl
PhysicalDataSource(java.lang.String name, RecordDescriptor rd, ISBounds isb, AxisOrdering ordering, java.lang.String URL) Create a Physical data source from specs

Method Summary

Methods

Modifier and Type	Method and Description
protected java.nio.ByteBuffer	allocateBuffer(java.nio.ByteBuffer iobuf, long byteSize) This method (re)allocates the argument I/O byte buffer to the byte size given by the second argument.
protected boolean	allocateBuffer(int size) Allocate buffer 0.
protected boolean	allocateBuffer(int buf, ISBounds queryBounds) This method allocates a proper I/O bytebuffer for the queryBounds passed in as an argument.
protected boolean	allocateBuffer(int buf, ISBounds queryBounds, int unitSize) This method allocates a proper I/O bytebuffer for the queryBounds passed in as an argument.
protected boolean	allocateBuffer(int buf, ISBounds gridSpace, ISBounds queryBounds, int unitSize) This method allocates a proper I/O bytebuffer for the specified grid space domain (gridBounds) and queryBounds passed in as arguments.
protected boolean	allocateBuffer(int buf, long byteSize) This method (re)allocates the specified I/O byte buffer to the byte size given by the second argument.
protected boolean	allocateBuffer(ISBounds bounds) Allocate buffer 0.
protected boolean	allocateBuffer(ISBounds bounds, int unitSize) Allocate buffer 0.
protected boolean	allocateBuffer(ISBounds grid, ISBounds bounds, int unitSize) Allocate buffer 0.
void	deactivate() The DataSource will release any resources used during the readling or processing of data.
void	deleteBuffer() Delete all byte buffers used for data storage for this DataSource and garbage collect.
protected void	deleteBuffer(int buf) delete a byte buffer and garbage collect.
void	deleteBuffers() Delete all byte buffers used for data storage for this DataSource and garbage collect.
boolean	equals(java.lang.Object o) Returns true if this PhysicalDataSource has the same URL and AxisOrdering as the argument PhsyicalDataSource, as well as meeting inherited criteria for equality.
protected java.nio.ByteBuffer	getBuffer() Return a reference to byte buffer 0, if there is only one.
protected java.nio.ByteBuffer	getBuffer(int i) Return a reference to the specified byte buffer
int	getBufferSize() get the current length, in bytes, of the buffer used directly with file I/O operations.
int	getBufferSize(int bufNum) get the current length, in bytes, of the buffer used directly with file I/O operations.
java.nio.ByteOrder	getByteOrder() Return the byteOrder of this binary file
java.lang.String	getDataFileString() Return a copy of the URL for this datasource.
int	getMaxBufferSize() get the maximum length, in bytes, of the buffer used directly with file I/O operations.
long	getReadCounter()
long	getReadLength()
java.lang.String	getURL() Return a copy of the URL for this datasource.
protected void	initializeBuffers(int num) Initialize buffers for i/o
boolean	isBigEndian() Returns true if the binary file is big-endian format or little endian
static void	main(java.lang.String[] args)
protected long	makeBestBuffer(ISBounds spaceBounds, ISBounds queryBounds, int unitSize) this method computes the best i/o buffer size.
void	resetReadCounter()
protected void	setBufferByteOrder(java.nio.ByteBuffer buffer) set the byte order for the given buffer
void	setByteOrder(java.nio.ByteOrder newOrder) set the byteOrder of this binary file.
void	setMaxBufferSize(int numBytes) set the maximum length, in bytes, of the buffer used directly with file I/O operations.
void	setMaxGapSize(long gap) gap means the number of extra data, the datasource willing to read to elimilate a read operation.
void	subblock(DataBlock dstBlock, ISBounds dstBounds, ISBounds srcBounds) Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds.
void	subblock(DataBlock dstBlock, ISBounds dstBounds, ISBounds srcBounds, FieldIDMapper mp) Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds.
void	subblock(DataBlock dstBlock, ISBounds dstBounds, ISBounds srcBounds, RecordSpec recSpec) Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds.
java.lang.String	toString() Return a String representing the object.
java.lang.String	toString(java.lang.String indent) Return a String representing the object indented by the argument string.

Methods inherited from class edu.unh.sdb.datasource.DataSource

activate, checkActivated, clearCache, copyBestID, create, create, create, createRemote, createRemote, createRemoteClient, createRemoteClient, createRemoteDataSource, createRemoteDataSource, createRemoteDataSourceUDP, createRemoteUDP, createRemoteUDP, dumpData, getFDLDescriptor, getMetadataCollection, getName, getStatistics, getValues, isActivated, setMetadataCollection

Methods inherited from class edu.unh.sdb.datasource.DataCollection

byteSize, computeOffset, computeOffset, copyBounds, createDataBlock, createDataBlock, createDataBlock, createDatum, createDatum, createDatum, createDatum, createDatum, createDatum, datum, datum, datum, datum, datum, datum, datum, dim, getAttributeNames, getAxisOrdering, getAxisWeight, getAxisWeights, getBounds, getByte, getBytes, getBytes, getBytes, getBytes, getBytes, getBytes, getBytesByAttribute, getBytesByAttribute, getBytesByPoint, getBytesByPoint, getDataOrder, getDimensionsArray, getDouble, getDoubles, getDoubles, getDoubles, getDoubles, getDoubles, getDoubles, getDoublesByAttribute, getDoublesByAttribute, getDoublesByPoint, getDoublesByPoint, getFloat, getFloats, getFloats, getFloats, getFloats, getFloats, getFloats, getFloatsByAttribute, getFloatsByAttribute, getFloatsByPoint, getFloatsByPoint, getInt, getInts, getInts, getInts, getInts, getInts, getInts, getIntsByAttribute, getIntsByAttribute, getIntsByPoint, getIntsByPoint, getLowerBoundsArray, getNumAttributes, getRecordDescriptor, getShort, getShorts, getShorts, getShorts, getShorts, getShorts, getShorts, getShortsByAttribute, getShortsByAttribute, getShortsByPoint, getShortsByPoint, getStorageType, getUpperBoundsArray, getValues, getValuesByAttribute, getValuesByPoint, longVolume, setAxisOrdering, setBounds, setRecordDescriptor, size, subblock, subblock, subblock, subblock, subblock, subblock, subblock, subblock, subblock, useBytesOnly, volume

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

BIG_ENDIAN

public static final java.nio.ByteOrder BIG_ENDIAN

LITTLE_ENDIAN

public static final java.nio.ByteOrder LITTLE_ENDIAN

endian

protected java.nio.ByteOrder endian

directMemoryLimit

public static long directMemoryLimit

maxBufferSize

protected long maxBufferSize

url

protected java.lang.String url

bufferDimension

protected IntegerIndexSpaceID bufferDimension

bufferBounds

protected ISBounds bufferBounds

bufferSteps

protected int[] bufferSteps

maxGapSize

protected long maxGapSize

byteBuffers

protected java.nio.ByteBuffer[] byteBuffers

readCounter

protected long readCounter

readLength

protected long readLength

Constructor Detail

PhysicalDataSource

public PhysicalDataSource(java.lang.String name,
                  FDLDescriptor fdl)

Create a PhysicalDataSource from an fdl

Parameters:

name -

fdl -

PhysicalDataSource

public PhysicalDataSource(java.lang.String name,
                  RecordDescriptor rd,
                  ISBounds isb,
                  AxisOrdering ordering,
                  java.lang.String URL)

Create a Physical data source from specs

Parameters:

name -

rd -

isb -

ordering -

URL -

Method Detail

initializeBuffers

protected void initializeBuffers(int num)

Initialize buffers for i/o

deactivate

public void deactivate()

The DataSource will release any resources used during the readling or processing of data. It is important that all child classes call super.deactivate()

Overrides:

deactivate in class DataSource

setMaxGapSize

public void setMaxGapSize(long gap)

gap means the number of extra data, the datasource willing to read to elimilate a read operation. If the user does not call this method the default gap size is 0

isBigEndian

public boolean isBigEndian()

Returns true if the binary file is big-endian format or little endian

getByteOrder

public java.nio.ByteOrder getByteOrder()

Return the byteOrder of this binary file

setByteOrder

public void setByteOrder(java.nio.ByteOrder newOrder)

set the byteOrder of this binary file. Useful when you want to prevent Granite from byte swapping little endian data. This is done by telling Granite that the data is big endian even though it isn't.

setBufferByteOrder

protected void setBufferByteOrder(java.nio.ByteBuffer buffer)

set the byte order for the given buffer

setMaxBufferSize

public void setMaxBufferSize(int numBytes)

set the maximum length, in bytes, of the buffer used directly with file I/O operations.

getMaxBufferSize

public int getMaxBufferSize()

get the maximum length, in bytes, of the buffer used directly with file I/O operations.

getBufferSize

public int getBufferSize()

get the current length, in bytes, of the buffer used directly with file I/O operations.

getBufferSize

public int getBufferSize(int bufNum)

get the current length, in bytes, of the buffer used directly with file I/O operations.

allocateBuffer

protected boolean allocateBuffer(int buf,
                     long byteSize)

This method (re)allocates the specified I/O byte buffer to the byte size given by the second argument. If the size of the existing buffer is correct, it is simply cleared. Otherwise, a new ByteBuffer is allocated. This method returns true if a buffer of the requested size was allocated and false otherwise. (rdb 03/2012: called from this class twice and from AttributeOrderBFDS twice)

allocateBuffer

protected java.nio.ByteBuffer allocateBuffer(java.nio.ByteBuffer iobuf,
                                 long byteSize)

This method (re)allocates the argument I/O byte buffer to the byte size given by the second argument. If the size of the existing buffer is correct, it is simply cleared. Otherwise, a new ByteBuffer is allocated. First arg is allowed to be null. (rdb 03/2012: not used)

allocateBuffer

protected final boolean allocateBuffer(int buf,
                     ISBounds queryBounds)

This method allocates a proper I/O bytebuffer for the queryBounds passed in as an argument. The query bounds is measured in grid Space. If the existing ByteBuffer is not the right size a new ByteBuffer is created. This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: called from this class)

allocateBuffer

protected final boolean allocateBuffer(int buf,
                     ISBounds queryBounds,
                     int unitSize)

This method allocates a proper I/O bytebuffer for the queryBounds passed in as an argument. The query bounds is measured in grid Space. The third argument represents the size needed for each point in the bounds. If the existing ByteBuffer is not the right size a new ByteBuffer is created. This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: called from this class)

allocateBuffer

protected boolean allocateBuffer(int buf,
                     ISBounds gridSpace,
                     ISBounds queryBounds,
                     int unitSize)

This method allocates a proper I/O bytebuffer for the specified grid space domain (gridBounds) and queryBounds passed in as arguments. The query bounds is measured in grid Space. The third argument represents the size needed for each point in the bounds. If the existing ByteBuffer is not the right size a new ByteBuffer is created. This is especially useful for allocating buffers by chunking. This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: called 3 times from this class)

allocateBuffer

protected final boolean allocateBuffer(int size)

Allocate buffer 0. This is a convenience function for most DataSources This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: called by ChunkedDataSource, PointOrderBFDS, SparseDataSource)

allocateBuffer

protected final boolean allocateBuffer(ISBounds bounds)

Allocate buffer 0. This is a convenience function for most DataSources This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: called by PointOrderBFDS)

allocateBuffer

protected final boolean allocateBuffer(ISBounds bounds,
                     int unitSize)

Allocate buffer 0. This is a convenience function for most DataSources This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: not used)

allocateBuffer

protected final boolean allocateBuffer(ISBounds grid,
                     ISBounds bounds,
                     int unitSize)

Allocate buffer 0. This is a convenience function for ChunkedDataSource This method returns true if a buffer long enough to contain a complete rod of the queryBounds was allocated, and false otherwise. (rdb 03/2012: called by ChunkedDataSource)

deleteBuffer

protected void deleteBuffer(int buf)

delete a byte buffer and garbage collect.

deleteBuffers

public void deleteBuffers()

Delete all byte buffers used for data storage for this DataSource and garbage collect.

deleteBuffer

public void deleteBuffer()

Delete all byte buffers used for data storage for this DataSource and garbage collect. Convenience name for 1-buffer datasources.

getBuffer

protected final java.nio.ByteBuffer getBuffer(int i)

Return a reference to the specified byte buffer

getBuffer

protected final java.nio.ByteBuffer getBuffer()

Return a reference to byte buffer 0, if there is only one. This is a convenience function for most DataSources

resetReadCounter

public void resetReadCounter()

getReadCounter

public long getReadCounter()

getReadLength

public long getReadLength()

subblock

public void subblock(DataBlock dstBlock,
            ISBounds dstBounds,
            ISBounds srcBounds)

Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds. The BasicBlock is assumed to be created and a valid data array is already allocated. NOTE: dstBlock can be BasicBlock or CompositeBlock. data is copied from file to BasicBlock or CompositeBlock based on its data type.

Specified by:

subblock in interface Indexable

Specified by:

subblock in class DataCollection

subblock

public void subblock(DataBlock dstBlock,
            ISBounds dstBounds,
            ISBounds srcBounds,
            RecordSpec recSpec)

Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds. The RecordSpec argument specifies a subset of the DataSource attributes to be written into the DataBlock datums. Note that the DataBlock is assumed to already have a valid data array. the DataBlock can be BasicBlock or CompositeBlock

Specified by:

subblock in interface Indexable

Overrides:

subblock in class DataCollection

subblock

public void subblock(DataBlock dstBlock,
            ISBounds dstBounds,
            ISBounds srcBounds,
            FieldIDMapper mp)

Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds. The FieldIDMapper argument determines the mapping between datums in the datasource and datums in the DataBlock. Note that the DataBlock is assumed to already have a valid data array. the dataBlock can be basicBlock or compositeBlock

Specified by:

subblock in interface Indexable

Specified by:

subblock in class DataCollection

getURL

public java.lang.String getURL()

Return a copy of the URL for this datasource.

getDataFileString

public java.lang.String getDataFileString()

Return a copy of the URL for this datasource.

toString

public java.lang.String toString()

Return a String representing the object.

Overrides:

toString in class DataSource

toString

public java.lang.String toString(java.lang.String indent)

Return a String representing the object indented by the argument string.

Overrides:

toString in class DataSource

equals

public boolean equals(java.lang.Object o)

Returns true if this PhysicalDataSource has the same URL and AxisOrdering as the argument PhsyicalDataSource, as well as meeting inherited criteria for equality.

Overrides:

equals in class DataSource

makeBestBuffer

protected long makeBestBuffer(ISBounds spaceBounds,
                  ISBounds queryBounds,
                  int unitSize)

this method computes the best i/o buffer size. As a default the buffer size will not be bigger than PhysicalDataSource.maxBufferSize because bigger buffers take a lot of memory and reduce the performance. This is intended to work for both "normal" and chunked data sets. The first argument is the bounds in units (partition space for chunked, datum space for regular). The second argument is the query within the spaceBounds; The third argument is the size of each unit in bytes. It returns the number of units included in the buffer It tries to create a buffer that extends as far as possible along as many less significant axes as will fit in the maximum as long as what will be read, but not needed (gap) does not get too large. In other words if the cost of reading the gap is larger than a subsequent seek over that gap, we'll stop increasing the buffer size. rdb: Complete rewrite 02/21/05

main

public static void main(java.lang.String[] args)