edu.unh.sdb.datasource

Class ChunkedDataSource

java.lang.Object

edu.unh.sdb.datasource.DataCollection

edu.unh.sdb.datasource.DataSource

edu.unh.sdb.datasource.PhysicalDataSource

edu.unh.sdb.datasource.ChunkedDataSource

All Implemented Interfaces:

Indexable

public class ChunkedDataSource
extends PhysicalDataSource

This class performs chunking I/O operations. The data file should have been reorganized according to the spacial chunking. We assume the disk block size is 4K(4086 bytes ) We also assume that the data file can be equally divided into a number of blocks. And the size of each block is a multiple of the disk block size NOTE: This class only works for BasicBlock so far.

Field Summary

Fields

Modifier and Type	Field and Description
protected RegularISPartitioning	chunkingPartitioning
protected ISBounds	partitioningBounds

Fields inherited from class edu.unh.sdb.datasource.PhysicalDataSource

BIG_ENDIAN, bufferBounds, bufferDimension, bufferSteps, byteBuffers, directMemoryLimit, endian, LITTLE_ENDIAN, maxBufferSize, maxGapSize, readCounter, readLength, url

Fields inherited from class edu.unh.sdb.datasource.DataSource

fdlDescriptor, isActivated, name, wsName

Fields inherited from class edu.unh.sdb.datasource.DataCollection

attributeOrder, axisOrdering, axisWeights, bounds, dataOrder, dimensionality, numAttributes, pointOrder, recordDesc

Constructor Summary

Constructors

Constructor and Description
ChunkedDataSource(java.lang.String name, FDLDescriptor fdl) Create a ChunkedDataSource from an fdl; In this case, the partitioning specifications are determined by the size of each partition, not the size of the partitioning space.
ChunkedDataSource(java.lang.String name, ISBounds partitioningBounds, ISBounds spaceBounds, java.lang.String fileName, AxisOrdering order, RecordDescriptor recordDesc) Creates a ChunkedDataSource object from 'fileName', how many chunks are on each direction are specified by 'partitioningBounds', the dimensions and placement of the volume given by 'spaceBounds', axis ordering 'order' and records described by 'recordDesc'.

Method Summary

Methods

Modifier and Type	Method and Description
void	activate() Open the binary data file for this datasource.
void	datum(Datum datum, IndexSpaceID dataId) load the data from the data source into the argument datum
void	datum(Datum datum, IndexSpaceID did, FieldIDMapper fmap) Return the Datum corresponding to the given IndexSpaceID.
void	deactivate() The DataSource will release any resources used during the readling or processing of data.
ISBounds	getChunkBounds() Return an ISBounds indicating the dimensions of a chunk.
int	getChunkByteSize() get the byte size of each chunk
IndexSpaceID	getChunkDimensions() get the dimensions of each chunk in grid space
ISBounds	getPartitionSpaceBounds() return the ISBounds of this chunkedDataSource in partitioned Space
protected void	loadChunk(long chunkIndex, Values data) Given a chunk index, read the data from the chunk into a Values object
static void	main(java.lang.String[] args)
int	numOfChunks() return number of chunks contained by the datasource
void	setOrder(java.nio.ByteBuffer buffer) set the byte order for the given buffer
void	subblock(DataBlock dstBlock, ISBounds dstBounds, ISBounds srcBounds, FieldIDMapper mp) Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds.

Methods inherited from class edu.unh.sdb.datasource.PhysicalDataSource

allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, allocateBuffer, deleteBuffer, deleteBuffer, deleteBuffers, equals, getBuffer, getBuffer, getBufferSize, getBufferSize, getByteOrder, getDataFileString, getMaxBufferSize, getReadCounter, getReadLength, getURL, initializeBuffers, isBigEndian, makeBestBuffer, resetReadCounter, setBufferByteOrder, setByteOrder, setMaxBufferSize, setMaxGapSize, subblock, subblock, toString, toString

Methods inherited from class edu.unh.sdb.datasource.DataSource

checkActivated, clearCache, copyBestID, create, create, create, createRemote, createRemote, createRemoteClient, createRemoteClient, createRemoteDataSource, createRemoteDataSource, createRemoteDataSourceUDP, createRemoteUDP, createRemoteUDP, dumpData, getFDLDescriptor, getMetadataCollection, getName, getStatistics, getValues, isActivated, setMetadataCollection

Methods inherited from class edu.unh.sdb.datasource.DataCollection

byteSize, computeOffset, computeOffset, copyBounds, createDataBlock, createDataBlock, createDataBlock, createDatum, createDatum, createDatum, createDatum, createDatum, createDatum, datum, datum, datum, datum, datum, dim, getAttributeNames, getAxisOrdering, getAxisWeight, getAxisWeights, getBounds, getByte, getBytes, getBytes, getBytes, getBytes, getBytes, getBytes, getBytesByAttribute, getBytesByAttribute, getBytesByPoint, getBytesByPoint, getDataOrder, getDimensionsArray, getDouble, getDoubles, getDoubles, getDoubles, getDoubles, getDoubles, getDoubles, getDoublesByAttribute, getDoublesByAttribute, getDoublesByPoint, getDoublesByPoint, getFloat, getFloats, getFloats, getFloats, getFloats, getFloats, getFloats, getFloatsByAttribute, getFloatsByAttribute, getFloatsByPoint, getFloatsByPoint, getInt, getInts, getInts, getInts, getInts, getInts, getInts, getIntsByAttribute, getIntsByAttribute, getIntsByPoint, getIntsByPoint, getLowerBoundsArray, getNumAttributes, getRecordDescriptor, getShort, getShorts, getShorts, getShorts, getShorts, getShorts, getShorts, getShortsByAttribute, getShortsByAttribute, getShortsByPoint, getShortsByPoint, getStorageType, getUpperBoundsArray, getValues, getValuesByAttribute, getValuesByPoint, longVolume, setAxisOrdering, setBounds, setRecordDescriptor, size, subblock, subblock, subblock, subblock, subblock, subblock, subblock, subblock, subblock, useBytesOnly, volume

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

partitioningBounds

protected ISBounds partitioningBounds

chunkingPartitioning

protected RegularISPartitioning chunkingPartitioning

Constructor Detail

ChunkedDataSource

public ChunkedDataSource(java.lang.String name,
                 FDLDescriptor fdl)

Create a ChunkedDataSource from an fdl; In this case, the partitioning specifications are determined by the size of each partition, not the size of the partitioning space. I.e., the user specifies that each partition will be defined, so we have to compute the bounds of the partitioning space (how many partitions are defined along each dimension).

Parameters:

name -

fdl -

ChunkedDataSource

public ChunkedDataSource(java.lang.String name,
                 ISBounds partitioningBounds,
                 ISBounds spaceBounds,
                 java.lang.String fileName,
                 AxisOrdering order,
                 RecordDescriptor recordDesc)

Creates a ChunkedDataSource object from 'fileName', how many chunks are on each direction are specified by 'partitioningBounds', the dimensions and placement of the volume given by 'spaceBounds', axis ordering 'order' and records described by 'recordDesc'. It seems that 'order' specifies axis ordering for both an individual chunk and for all chunks. This constructor must compute the "nominal" size of each partition; partitions on the borders may be smaller, but this feature is not completely supported right now. 'datum' method caches one chunk at a time.

Method Detail

getPartitionSpaceBounds

public ISBounds getPartitionSpaceBounds()

return the ISBounds of this chunkedDataSource in partitioned Space

activate

public void activate()

Open the binary data file for this datasource.

Overrides:

activate in class DataSource

deactivate

public void deactivate()

The DataSource will release any resources used during the readling or processing of data.

Overrides:

deactivate in class PhysicalDataSource

setOrder

public void setOrder(java.nio.ByteBuffer buffer)

set the byte order for the given buffer

getChunkByteSize

public int getChunkByteSize()

get the byte size of each chunk

getChunkDimensions

public IndexSpaceID getChunkDimensions()

get the dimensions of each chunk in grid space

getChunkBounds

public ISBounds getChunkBounds()

Return an ISBounds indicating the dimensions of a chunk.

datum

public void datum(Datum datum,
         IndexSpaceID dataId)

load the data from the data source into the argument datum

Specified by:

datum in interface Indexable

Overrides:

datum in class DataCollection

datum

public void datum(Datum datum,
         IndexSpaceID did,
         FieldIDMapper fmap)

Return the Datum corresponding to the given IndexSpaceID. The Datum will only contain attributes specified by the RecordSpec argument.

Specified by:

datum in interface Indexable

Specified by:

datum in class DataCollection

loadChunk

protected void loadChunk(long chunkIndex,
             Values data)

Given a chunk index, read the data from the chunk into a Values object

numOfChunks

public int numOfChunks()

return number of chunks contained by the datasource

main

public static void main(java.lang.String[] args)

subblock

public void subblock(DataBlock dstBlock,
            ISBounds dstBounds,
            ISBounds srcBounds,
            FieldIDMapper mp)

Reads a subblock defined by srcBounds into the region of the Datablock argument defined by dstBounds. The FieldIDMapper argument determines the mapping between datums in the datasource and datums in the DataBlock. Note that the DataBlock is assumed to already have a valid data array. the dataBlock can be basicBlock or compositeBlock

Specified by:

subblock in interface Indexable

Overrides:

subblock in class PhysicalDataSource