Class InfrastructureAnalyzer
- java.lang.Object
- 
- org.apache.sysds.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer
 
- 
 public class InfrastructureAnalyzer extends Object Central place for analyzing and obtaining static infrastructure properties such as memory and number of logical processors.
- 
- 
Field SummaryFields Modifier and Type Field Description static longDEFAULT_JVM_SIZEstatic StringMR_FRAMEWORK_NAMEstatic StringMR_JOBTRACKER_ADDRESS
 - 
Constructor SummaryConstructors Constructor Description InfrastructureAnalyzer()
 - 
Method SummaryAll Methods Static Methods Concrete Methods Modifier and Type Method Description static longextractMaxMemoryOpt(String javaOpts)static intgetCkMaxCP()Gets the maximum local parallelism constraint.static intgetCkMaxMR()Gets the maximum remote parallelism constraintstatic doublegetClusterUtilization(boolean mapOnly)Gets the fraction of running map/reduce tasks to existing map/reduce task slots.static longgetCmMax()Gets the maximum memory constraint [in bytes].static longgetHDFSBlockSize()Gets the HDFS blocksize of the used cluster in bytes.static longgetLocalMaxMemory()Gets the maximum memory [in bytes] of the current JVM.static doublegetLocalMaxMemoryFraction()static intgetLocalParallelism()Gets the number of logical processors of the current node, including hyper-threading if enabled.static intgetRemoteParallelMapTasks()Gets the number of remote parallel map slots.static intgetRemoteParallelNodes()Gets the number of cluster nodes (number of tasktrackers).static intgetRemoteParallelReduceTasks()Gets the total number of remote parallel reduce slots.static booleanisLocalMode()static booleanisLocalMode(org.apache.hadoop.mapred.JobConf job)static voidsetLocalMaxMemory(long localMem)static voidsetMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)static voidsetRemoteParallelMapTasks(int pmap)static voidsetRemoteParallelReduceTasks(int preduce)
 
- 
- 
- 
Field Detail- 
DEFAULT_JVM_SIZEpublic static final long DEFAULT_JVM_SIZE - See Also:
- Constant Field Values
 
 - 
MR_FRAMEWORK_NAMEpublic static final String MR_FRAMEWORK_NAME - See Also:
- Constant Field Values
 
 - 
MR_JOBTRACKER_ADDRESSpublic static final String MR_JOBTRACKER_ADDRESS - See Also:
- Constant Field Values
 
 
- 
 - 
Method Detail- 
getLocalParallelismpublic static int getLocalParallelism() Gets the number of logical processors of the current node, including hyper-threading if enabled.- Returns:
- number of local processors of the current node
 
 - 
getRemoteParallelNodespublic static int getRemoteParallelNodes() Gets the number of cluster nodes (number of tasktrackers). If multiple tasktracker are started per node, each tasktracker is viewed as individual node.- Returns:
- number of cluster nodes
 
 - 
getRemoteParallelMapTaskspublic static int getRemoteParallelMapTasks() Gets the number of remote parallel map slots.- Returns:
- number of remote parallel map tasks
 
 - 
setRemoteParallelMapTaskspublic static void setRemoteParallelMapTasks(int pmap) 
 - 
getRemoteParallelReduceTaskspublic static int getRemoteParallelReduceTasks() Gets the total number of remote parallel reduce slots.- Returns:
- number of remote parallel reduce tasks
 
 - 
setRemoteParallelReduceTaskspublic static void setRemoteParallelReduceTasks(int preduce) 
 - 
getLocalMaxMemorypublic static long getLocalMaxMemory() Gets the maximum memory [in bytes] of the current JVM.- Returns:
- maximum memory of the current JVM
 
 - 
setLocalMaxMemorypublic static void setLocalMaxMemory(long localMem) 
 - 
getLocalMaxMemoryFractionpublic static double getLocalMaxMemoryFraction() 
 - 
isLocalModepublic static boolean isLocalMode() 
 - 
isLocalModepublic static boolean isLocalMode(org.apache.hadoop.mapred.JobConf job) 
 - 
getCkMaxCPpublic static int getCkMaxCP() Gets the maximum local parallelism constraint.- Returns:
- maximum local parallelism constraint
 
 - 
getCkMaxMRpublic static int getCkMaxMR() Gets the maximum remote parallelism constraint- Returns:
- maximum remote parallelism constraint
 
 - 
getCmMaxpublic static long getCmMax() Gets the maximum memory constraint [in bytes].- Returns:
- maximum memory constraint
 
 - 
getHDFSBlockSizepublic static long getHDFSBlockSize() Gets the HDFS blocksize of the used cluster in bytes.- Returns:
- HDFS block size
 
 - 
extractMaxMemoryOptpublic static long extractMaxMemoryOpt(String javaOpts) 
 - 
setMaxMemoryOptpublic static void setMaxMemoryOpt(org.apache.hadoop.mapred.JobConf job, String key, long bytes)
 - 
getClusterUtilizationpublic static double getClusterUtilization(boolean mapOnly) throws IOExceptionGets the fraction of running map/reduce tasks to existing map/reduce task slots. NOTE: on YARN the number of slots is a spurious indicator because containers are purely scheduled based on memory.- Parameters:
- mapOnly- if true, only look at map tasks
- Returns:
- cluster utilization (current / capacity)
- Throws:
- IOException- if IOException occurs
 
 
- 
 
-