Class PartitionedBroadcast<T extends CacheBlock>
- java.lang.Object
- 
- org.apache.sysds.runtime.instructions.spark.data.PartitionedBroadcast<T>
 
- 
- All Implemented Interfaces:
- Serializable
 
 public class PartitionedBroadcast<T extends CacheBlock> extends Object implements Serializable This class is a wrapper around an array of broadcasts of partitioned matrix/frame blocks, which is required due to 2GB limitations of Spark's broadcast handling. Without this partitioning ofBroadcast<PartitionedBlock>intoBroadcast<PartitionedBlock>[], we got java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE issue. Despite various jiras, this issue still showed up in Spark 2.1.- See Also:
- Serialized Form
 
- 
- 
Constructor SummaryConstructors Constructor Description PartitionedBroadcast()PartitionedBroadcast(org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] broadcasts, DataCharacteristics dc)
 - 
Method SummaryAll Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description static intcomputeBlocksPerPartition(long[] dims, int blen)static intcomputeBlocksPerPartition(long rlen, long clen, long blen)voiddestroy()This method cleanups all underlying broadcasts of a partitioned broadcast, by forward the calls to SparkExecutionContext.cleanupBroadcastVariable.TgetBlock(int[] ix)TgetBlock(int rowIndex, int colIndex)org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[]getBroadcasts()DataCharacteristicsgetDataCharacteristics()longgetNumCols()intgetNumColumnBlocks()intgetNumRowBlocks()longgetNumRows()Tslice(long rl, long ru, long cl, long cu, T block)Utility for slice operations over partitioned matrices, where the index range can cover multiple blocks.
 
- 
- 
- 
Constructor Detail- 
PartitionedBroadcastpublic PartitionedBroadcast() 
 - 
PartitionedBroadcastpublic PartitionedBroadcast(org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] broadcasts, DataCharacteristics dc) 
 
- 
 - 
Method Detail- 
getBroadcastspublic org.apache.spark.broadcast.Broadcast<PartitionedBlock<T>>[] getBroadcasts() 
 - 
getNumRowspublic long getNumRows() 
 - 
getNumColspublic long getNumCols() 
 - 
getNumRowBlockspublic int getNumRowBlocks() 
 - 
getNumColumnBlockspublic int getNumColumnBlocks() 
 - 
getDataCharacteristicspublic DataCharacteristics getDataCharacteristics() 
 - 
computeBlocksPerPartitionpublic static int computeBlocksPerPartition(long rlen, long clen, long blen)
 - 
computeBlocksPerPartitionpublic static int computeBlocksPerPartition(long[] dims, int blen)
 - 
getBlockpublic T getBlock(int rowIndex, int colIndex) 
 - 
getBlockpublic T getBlock(int[] ix) 
 - 
slicepublic T slice(long rl, long ru, long cl, long cu, T block) Utility for slice operations over partitioned matrices, where the index range can cover multiple blocks. The result is always a single result matrix block. All semantics are equivalent to the core matrix block slice operations.- Parameters:
- rl- row lower bound
- ru- row upper bound
- cl- column lower bound
- cu- column upper bound
- block- block object
- Returns:
- block object
 
 - 
destroypublic void destroy() This method cleanups all underlying broadcasts of a partitioned broadcast, by forward the calls to SparkExecutionContext.cleanupBroadcastVariable.
 
- 
 
-