Package org.apache.sysds.api.mlcontext
Class MLContextUtil
- java.lang.Object
- 
- org.apache.sysds.api.mlcontext.MLContextUtil
 
- 
 public final class MLContextUtil extends Object Utility class containing methods for working with the MLContext API.
- 
- 
Field SummaryFields Modifier and Type Field Description static Class[]ALL_SUPPORTED_DATA_TYPESAll data types supported by the MLContext API.static Class[]BASIC_DATA_TYPESBasic data types supported by the MLContext API.static StringBUILD_TIME_NOT_AVAILABLEBuild time not available message.static Class[]COMPLEX_DATA_TYPESComplex data types supported by the MLContext API.static StringVERSION_NOT_AVAILABLEVersion not available message.
 - 
Constructor SummaryConstructors Constructor Description MLContextUtil()
 - 
Method SummaryAll Methods Static Methods Concrete Methods Modifier and Type Method Description static voidcheckInputParameterType(String parameterName, Object parameterValue)Verify that the type of input parameter value is supported.static voidcheckInputValueType(String name, Object value)Verify that the type of input value is supported.static voidcheckInputValueTypes(Map<String,Object> inputs)Verify that the types of input values are supported.static Map<String,String>convertInputParametersForParser(Map<String,Object> basicInputParameterMap)Converts non-string basic input parameter values to strings to pass to the parser.static DataconvertInputType(String parameterName, Object parameterValue)Convert input types to internal SystemDS representationsstatic DataconvertInputType(String parameterName, Object parameterValue, Metadata metadata)Convert input types to internal SystemDS representationsstatic voiddeleteRemoveVariableInstructions(Program progam)Delete the 'remove variable' instructions from a runtime program.static StringdetermineOutputTypeAsString(LocalVariableMap symbolTable, String outputName)Obtain a symbol table output type as a Stringstatic StringdisplayInputs(String name, Map<String,Object> map, LocalVariableMap symbolTable)Obtain a display of script inputs.static StringdisplayMap(String mapName, Map<String,Object> map)Display the keys and values in a Mapstatic StringdisplayOutputs(String name, Set<String> outputNames, LocalVariableMap symbolTable)Obtain a display of the script outputs.static StringdisplayOutputs(Set<String> outputNames, LocalVariableMap symbolTable)Obtain a display of the script outputs.static StringdisplaySet(String setName, Set<String> set)Display the values in a Setstatic StringdisplaySymbolTable(String name, LocalVariableMap symbolTable)Display the keys and values in the symbol tablestatic StringdisplaySymbolTable(LocalVariableMap symbolTable)Display the keys and values in the symbol tablestatic booleandoesDataFrameLookLikeMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df)Examine the DataFrame schema to determine whether the data appears to be a matrix.static booleandoesSymbolTableContainFrameObject(LocalVariableMap symbolTable, String variableName)Determine if the symbol table contains a FrameObject with the given variable name.static booleandoesSymbolTableContainMatrixObject(LocalVariableMap symbolTable, String variableName)Determine if the symbol table contains a MatrixObject with the given variable name.static StringgetBasicTypeString(Object object)Obtain the SystemDS scalar value type string equivalent of an accepted basic type (Integer, Boolean, Double, String)static org.apache.spark.api.java.JavaSparkContextgetJavaSparkContext(MLContext mlContext)Obtain the Java Spark Contextstatic org.apache.spark.api.java.JavaSparkContextgetJavaSparkContextFromProxy()Obtain the Java Spark Context from the MLContextProxystatic org.apache.spark.SparkContextgetSparkContext(MLContext mlContext)Obtain the Spark Contextstatic org.apache.spark.SparkContextgetSparkContextFromProxy()Obtain the Spark Context from the MLContextProxystatic org.apache.spark.sql.SparkSessiongetSparkSessionFromProxy()Obtain the Spark Session from the MLContextProxystatic booleanisBasicType(Object object)Is the object one of the supported basic data types? (Integer, Boolean, Double, String)static booleanisComplexType(Object object)Is the object one of the supported complex data types? (JavaRDD, RDD, DataFrame, Matrix, double[][], MatrixBlock, URL)static booleanisCSVLineAllNumbers(String line)If no metadata is supplied for an RDD or JavaRDD, this method can be used to determine whether the data appears to be matrix (or a frame)static booleanisSparkVersionSupported(String sparkVersion, String minimumRecommendedSparkVersion)Determine whether the Spark version is supported.static StringquotedString(String str)Return a double-quoted string with inner single and double quotes escaped.static voidsetCompilerConfig()Set SystemDS compiler configuration properties for MLContextstatic voidsetConfig(String configFilePath)Set SystemDS configuration properties based on a configuration file.static voidsetDefaultConfig()Set default SystemDS configuration properties.static voidverifySparkVersionSupported(org.apache.spark.sql.SparkSession spark)Check that the Spark version is supported.static StringwelcomeMessage()The SystemDS welcome message
 
- 
- 
- 
Field Detail- 
VERSION_NOT_AVAILABLEpublic static final String VERSION_NOT_AVAILABLE Version not available message.- See Also:
- Constant Field Values
 
 - 
BUILD_TIME_NOT_AVAILABLEpublic static final String BUILD_TIME_NOT_AVAILABLE Build time not available message.- See Also:
- Constant Field Values
 
 - 
BASIC_DATA_TYPESpublic static final Class[] BASIC_DATA_TYPES Basic data types supported by the MLContext API.
 - 
COMPLEX_DATA_TYPESpublic static final Class[] COMPLEX_DATA_TYPES Complex data types supported by the MLContext API.
 - 
ALL_SUPPORTED_DATA_TYPESpublic static final Class[] ALL_SUPPORTED_DATA_TYPES All data types supported by the MLContext API.
 
- 
 - 
Method Detail- 
isSparkVersionSupportedpublic static boolean isSparkVersionSupported(String sparkVersion, String minimumRecommendedSparkVersion) Determine whether the Spark version is supported.- Parameters:
- sparkVersion- Spark version string (ie, "1.5.0").
- minimumRecommendedSparkVersion- Minimum recommended Spark version string (ie, "2.1.0").
- Returns:
- trueif Spark version supported; otherwise- false.
 
 - 
verifySparkVersionSupportedpublic static void verifySparkVersionSupported(org.apache.spark.sql.SparkSession spark) Check that the Spark version is supported. If it isn't supported, throw an MLContextException.- Parameters:
- spark- SparkSession
- Throws:
- MLContextException- thrown if Spark version isn't supported
 
 - 
setDefaultConfigpublic static void setDefaultConfig() Set default SystemDS configuration properties.
 - 
setConfigpublic static void setConfig(String configFilePath) Set SystemDS configuration properties based on a configuration file.- Parameters:
- configFilePath- Path to configuration file.
- Throws:
- MLContextException- if configuration file was not found or a parse exception occurred
 
 - 
setCompilerConfigpublic static void setCompilerConfig() Set SystemDS compiler configuration properties for MLContext
 - 
checkInputValueTypespublic static void checkInputValueTypes(Map<String,Object> inputs) Verify that the types of input values are supported.- Parameters:
- inputs- Map of String/Object pairs
- Throws:
- MLContextException- if an input value type is not supported
 
 - 
checkInputValueTypepublic static void checkInputValueType(String name, Object value) Verify that the type of input value is supported.- Parameters:
- name- The name of the input
- value- The value of the input
- Throws:
- MLContextException- if the input value type is not supported
 
 - 
checkInputParameterTypepublic static void checkInputParameterType(String parameterName, Object parameterValue) Verify that the type of input parameter value is supported.- Parameters:
- parameterName- The name of the input parameter
- parameterValue- The value of the input parameter
- Throws:
- MLContextException- if the input parameter value type is not supported
 
 - 
isBasicTypepublic static boolean isBasicType(Object object) Is the object one of the supported basic data types? (Integer, Boolean, Double, String)- Parameters:
- object- the object type to be examined
- Returns:
- trueif type is a basic data type; otherwise- false.
 
 - 
getBasicTypeStringpublic static String getBasicTypeString(Object object) Obtain the SystemDS scalar value type string equivalent of an accepted basic type (Integer, Boolean, Double, String)- Parameters:
- object- the object type to be examined
- Returns:
- a String representing the type as a SystemDS scalar value type
 
 - 
isComplexTypepublic static boolean isComplexType(Object object) Is the object one of the supported complex data types? (JavaRDD, RDD, DataFrame, Matrix, double[][], MatrixBlock, URL)- Parameters:
- object- the object type to be examined
- Returns:
- trueif type is a complex data type; otherwise- false.
 
 - 
convertInputParametersForParserpublic static Map<String,String> convertInputParametersForParser(Map<String,Object> basicInputParameterMap) Converts non-string basic input parameter values to strings to pass to the parser.- Parameters:
- basicInputParameterMap- map of input parameters
- Returns:
- map of String/String name/value pairs
 
 - 
convertInputTypepublic static Data convertInputType(String parameterName, Object parameterValue) Convert input types to internal SystemDS representations- Parameters:
- parameterName- The name of the input parameter
- parameterValue- The value of the input parameter
- Returns:
- input in SystemDS data representation
 
 - 
convertInputTypepublic static Data convertInputType(String parameterName, Object parameterValue, Metadata metadata) Convert input types to internal SystemDS representations- Parameters:
- parameterName- The name of the input parameter
- parameterValue- The value of the input parameter
- metadata- matrix/frame metadata
- Returns:
- input in SystemDS data representation
 
 - 
isCSVLineAllNumberspublic static boolean isCSVLineAllNumbers(String line) If no metadata is supplied for an RDD or JavaRDD, this method can be used to determine whether the data appears to be matrix (or a frame)- Parameters:
- line- a line of the RDD
- Returns:
- trueif all the csv-separated values are numbers,- falseotherwise
 
 - 
doesDataFrameLookLikeMatrixpublic static boolean doesDataFrameLookLikeMatrix(org.apache.spark.sql.Dataset<org.apache.spark.sql.Row> df) Examine the DataFrame schema to determine whether the data appears to be a matrix.- Parameters:
- df- the DataFrame
- Returns:
- trueif the DataFrame appears to be a matrix,- falseotherwise
 
 - 
quotedStringpublic static String quotedString(String str) Return a double-quoted string with inner single and double quotes escaped.- Parameters:
- str- the original string
- Returns:
- double-quoted string with inner single and double quotes escaped
 
 - 
displayMappublic static String displayMap(String mapName, Map<String,Object> map) Display the keys and values in a Map- Parameters:
- mapName- the name of the map
- map- Map of String keys and Object values
- Returns:
- the keys and values in the Map as a String
 
 - 
displaySetpublic static String displaySet(String setName, Set<String> set) Display the values in a Set- Parameters:
- setName- the name of the Set
- set- Set of String values
- Returns:
- the values in the Set as a String
 
 - 
displaySymbolTablepublic static String displaySymbolTable(String name, LocalVariableMap symbolTable) Display the keys and values in the symbol table- Parameters:
- name- the name of the symbol table
- symbolTable- the LocalVariableMap
- Returns:
- the keys and values in the symbol table as a String
 
 - 
displaySymbolTablepublic static String displaySymbolTable(LocalVariableMap symbolTable) Display the keys and values in the symbol table- Parameters:
- symbolTable- the LocalVariableMap
- Returns:
- the keys and values in the symbol table as a String
 
 - 
determineOutputTypeAsStringpublic static String determineOutputTypeAsString(LocalVariableMap symbolTable, String outputName) Obtain a symbol table output type as a String- Parameters:
- symbolTable- the symbol table
- outputName- the name of the output variable
- Returns:
- the symbol table output type for a variable as a String
 
 - 
displayInputspublic static String displayInputs(String name, Map<String,Object> map, LocalVariableMap symbolTable) Obtain a display of script inputs.- Parameters:
- name- the title to display for the inputs
- map- the map of inputs
- symbolTable- the symbol table
- Returns:
- the script inputs represented as a String
 
 - 
displayOutputspublic static String displayOutputs(String name, Set<String> outputNames, LocalVariableMap symbolTable) Obtain a display of the script outputs.- Parameters:
- name- the title to display for the outputs
- outputNames- the names of the output variables
- symbolTable- the symbol table
- Returns:
- the script outputs represented as a String
 
 - 
displayOutputspublic static String displayOutputs(Set<String> outputNames, LocalVariableMap symbolTable) Obtain a display of the script outputs.- Parameters:
- outputNames- the names of the output variables
- symbolTable- the symbol table
- Returns:
- the script outputs represented as a String
 
 - 
welcomeMessagepublic static String welcomeMessage() The SystemDS welcome message- Returns:
- the SystemDS welcome message
 
 - 
getSparkContextpublic static org.apache.spark.SparkContext getSparkContext(MLContext mlContext) Obtain the Spark Context- Parameters:
- mlContext- the SystemDS MLContext
- Returns:
- the Spark Context
 
 - 
getJavaSparkContextpublic static org.apache.spark.api.java.JavaSparkContext getJavaSparkContext(MLContext mlContext) Obtain the Java Spark Context- Parameters:
- mlContext- the SystemDS MLContext
- Returns:
- the Java Spark Context
 
 - 
getSparkContextFromProxypublic static org.apache.spark.SparkContext getSparkContextFromProxy() Obtain the Spark Context from the MLContextProxy- Returns:
- the Spark Context
 
 - 
getJavaSparkContextFromProxypublic static org.apache.spark.api.java.JavaSparkContext getJavaSparkContextFromProxy() Obtain the Java Spark Context from the MLContextProxy- Returns:
- the Java Spark Context
 
 - 
getSparkSessionFromProxypublic static org.apache.spark.sql.SparkSession getSparkSessionFromProxy() Obtain the Spark Session from the MLContextProxy- Returns:
- the Spark Session
 
 - 
doesSymbolTableContainFrameObjectpublic static boolean doesSymbolTableContainFrameObject(LocalVariableMap symbolTable, String variableName) Determine if the symbol table contains a FrameObject with the given variable name.- Parameters:
- symbolTable- the LocalVariableMap
- variableName- the variable name
- Returns:
- trueif the variable in the symbol table is a FrameObject,- falseotherwise.
 
 - 
doesSymbolTableContainMatrixObjectpublic static boolean doesSymbolTableContainMatrixObject(LocalVariableMap symbolTable, String variableName) Determine if the symbol table contains a MatrixObject with the given variable name.- Parameters:
- symbolTable- the LocalVariableMap
- variableName- the variable name
- Returns:
- trueif the variable in the symbol table is a MatrixObject,- falseotherwise.
 
 - 
deleteRemoveVariableInstructionspublic static void deleteRemoveVariableInstructions(Program progam) Delete the 'remove variable' instructions from a runtime program.- Parameters:
- progam- runtime program
 
 
- 
 
-