Class RemoteParForUtils
- java.lang.Object
- 
- org.apache.sysds.runtime.controlprogram.parfor.RemoteParForUtils
 
- 
 public class RemoteParForUtils extends Object Common functionalities for parfor workers in MR jobs. Used by worker wrappers in mappers (base RemoteParFor) and reducers (fused data partitioning and parfor)
- 
- 
Constructor SummaryConstructors Constructor Description RemoteParForUtils()
 - 
Method SummaryAll Methods Static Methods Concrete Methods Modifier and Type Method Description static voidcleanupWorkingDirectories()Cleanup all temporary files created by this SystemDS process.static voidcleanupWorkingDirectoriesOnShutdown()Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.static voidexportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage)Export lineage for remote Spark parfor workers.static ArrayList<String>exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars)For remote Spark parfor workers.static Lineage[]getLineages(LocalVariableMap[] results)static LocalVariableMap[]getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG)static voidincrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)static voidsetupBufferPool(long workerID, boolean isLocal)Init and register-cleanup of buffer pool
 
- 
- 
- 
Method Detail- 
incrementParForMRCounterspublic static void incrementParForMRCounters(org.apache.hadoop.mapred.Reporter reporter, long deltaTasks, long deltaIterations)
 - 
exportResultVariablespublic static ArrayList<String> exportResultVariables(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars) throws IOException For remote Spark parfor workers. This is a simplified version compared to MR.- Parameters:
- workerID- worker id
- vars- local variable map
- resultVars- list of result variables
- Returns:
- list of result variables
- Throws:
- IOException- if IOException occurs
 
 - 
exportLineageItemspublic static void exportLineageItems(long workerID, LocalVariableMap vars, ArrayList<ParForStatementBlock.ResultVar> resultVars, Lineage lineage) throws IOExceptionExport lineage for remote Spark parfor workers.- Parameters:
- workerID- worker id
- vars- local variable map
- resultVars- list of result variables
- lineage- lineage object
- Throws:
- IOException- if IOException occurs
 
 - 
cleanupWorkingDirectoriespublic static void cleanupWorkingDirectories() Cleanup all temporary files created by this SystemDS process.
 - 
cleanupWorkingDirectoriesOnShutdownpublic static void cleanupWorkingDirectoriesOnShutdown() Cleanup all temporary files created by this SystemDS process, on shutdown via exit or interrupt.
 - 
getLineagespublic static Lineage[] getLineages(LocalVariableMap[] results) 
 - 
getResultspublic static LocalVariableMap[] getResults(List<scala.Tuple2<Long,String>> out, org.apache.commons.logging.Log LOG) 
 - 
setupBufferPoolpublic static void setupBufferPool(long workerID, boolean isLocal) throws IOExceptionInit and register-cleanup of buffer pool- Parameters:
- workerID- worker id
- isLocal- in local spark mode (single JVM)
- Throws:
- IOException- exception
 
 
- 
 
-