Package org.apache.sysds.api.jmlc
Class Connection
- java.lang.Object
- 
- org.apache.sysds.api.jmlc.Connection
 
- 
- All Implemented Interfaces:
- Closeable,- AutoCloseable
 
 public class Connection extends Object implements Closeable Interaction with SystemDS using the JMLC (Java Machine Learning Connector) API is initiated with aConnectionobject. The JMLC API is designed after JDBC. A DML script is precompiled by calling theprepareScript(String, String[], String[])method or theprepareScript(String, Map, String[], String[])method on theConnectionobject, which returns aPreparedScriptobject. Note that this is similar to calling aprepareStatementmethod on a JDBCConnectionobject.Following this, input variable data is passed to the script by calling the setFrame,setMatrix, andsetScalarmethods of thePreparedScriptobject. The script is executed viaPreparedScript'sexecuteScriptmethod, which returns aResultVariablesobject, which is similar to a JDBCResultSet. Data can be read from aResultVariablesobject by calling itsgetFrameandgetMatrixmethods.
- 
- 
Constructor SummaryConstructors Constructor Description Connection()Connection constructor, the starting point for any other JMLC API calls.Connection(CompilerConfig.ConfigType... cconfigs)Connection constructor, the starting point for any other JMLC API calls.Connection(DMLConfig dmlconfig)Connection constructor, the starting point for any other JMLC API calls.Connection(DMLConfig dmlconfig, CompilerConfig.ConfigType... cconfigs)Connection constructor, the starting point for any other JMLC API calls.
 - 
Method SummaryAll Methods Instance Methods Concrete Methods Modifier and Type Method Description voidclose()Close connection to SystemDS, which clears the thread-local DML and compiler configurations.double[][]convertToDoubleMatrix(InputStream input, int rows, int cols)Converts an input stream of a string matrix in textcell format into a dense double array.double[][]convertToDoubleMatrix(InputStream input, int rows, int cols, String format)Converts an input stream of a string matrix in csv or textcell format into a dense double array.double[][]convertToDoubleMatrix(String input, int rows, int cols)Converts an input string representation of a matrix in textcell format into a dense double array.double[][]convertToDoubleMatrix(String input, String meta)Converts an input string representation of a matrix in csv or textcell format into a dense double array.FrameBlockconvertToFrame(InputStream input, int rows, int cols)Converts an input stream of a string frame in textcell format into a frame block.FrameBlockconvertToFrame(InputStream input, int rows, int cols, String format)Converts an input stream of a frame in csv or textcell format into a frame block.FrameBlockconvertToFrame(InputStream input, String meta)Converts an input stream of a string frame in csv or textcell format into a frame block.FrameBlockconvertToFrame(String input, int rows, int cols)Converts an input string representation of a frame in textcell format into a frame block.FrameBlockconvertToFrame(String input, String meta)Converts an input string representation of a frame in csv or textcell format into a frame block.MatrixBlockconvertToMatrix(InputStream input, int rows, int cols)Converts an input stream of a string matrix in text format into a matrix block.MatrixBlockconvertToMatrix(InputStream input, int rows, int cols, String format)Converts an input stream of a string matrix in csv or text format into a matrix block.MatrixBlockconvertToMatrix(InputStream input, String meta)Converts an input stream of a string matrix in csv or textcell format into a matrix block.MatrixBlockconvertToMatrix(String input, int rows, int cols)Converts an input string representation of a matrix in textcell format into a matrix block.MatrixBlockconvertToMatrix(String input, String meta)Converts an input string representation of a matrix in csv or textcell format into a matrix block.String[][]convertToStringFrame(InputStream input, int rows, int cols)Converts an input stream of a string frame in textcell format into a dense string array.String[][]convertToStringFrame(InputStream input, int rows, int cols, String format)Converts an input stream of a string frame in csv or text format into a dense string array.String[][]convertToStringFrame(String input, int rows, int cols)Converts an input stream of a string frame in textcell format into a dense string array.String[][]convertToStringFrame(String input, String meta)Converts an input string representation of a frame in csv or textcell format into a dense string array.voidgatherMemStats(boolean stats)Sets a boolean flag indicating if memory profiling statistics should be gathered.PreparedScriptprepareScript(String script, String[] inputs, String[] outputs)Prepares (precompiles) a script and registers input and output variables.PreparedScriptprepareScript(String script, Map<String,String> args, String[] inputs, String[] outputs)Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.PreparedScriptprepareScript(String script, Map<String,String> nsscripts, Map<String,String> args, String[] inputs, String[] outputs)Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.double[][]readDoubleMatrix(String fname)Reads an input matrix in arbitrary format from HDFS into a dense double array.double[][]readDoubleMatrix(String fname, Types.FileFormat fmt, long rows, long cols, int blen, long nnz)Reads an input matrix in arbitrary format from HDFS into a dense double array.StringreadScript(String fname)Read a DML or PyDML file as a string.String[][]readStringFrame(String fname)Reads an input frame in arbitrary format from HDFS into a dense string array.String[][]readStringFrame(String fname, Types.FileFormat fmt, long rows, long cols)Reads an input frame in arbitrary format from HDFS into a dense string array.FrameBlockreadTransformMetaDataFromFile(String metapath)Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.FrameBlockreadTransformMetaDataFromFile(String spec, String metapath)Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.FrameBlockreadTransformMetaDataFromFile(String spec, String metapath, String colDelim)Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.FrameBlockreadTransformMetaDataFromPath(String metapath)Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.FrameBlockreadTransformMetaDataFromPath(String spec, String metapath)Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.FrameBlockreadTransformMetaDataFromPath(String spec, String metapath, String colDelim)Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.voidsetLineage(boolean lt)Sets a boolean flag indicating if lineage trace should be capturedvoidsetStatistics(boolean stats)Sets a boolean flag indicating if runtime statistics should be gathered Same behavior as in "MLContext.setStatistics()"
 
- 
- 
- 
Constructor Detail- 
Connectionpublic Connection() Connection constructor, the starting point for any other JMLC API calls.
 - 
Connectionpublic Connection(CompilerConfig.ConfigType... cconfigs) Connection constructor, the starting point for any other JMLC API calls. This variant allows to enable a set of boolean compiler configurations.- Parameters:
- cconfigs- one or many boolean compiler configurations to enable.
 
 - 
Connectionpublic Connection(DMLConfig dmlconfig, CompilerConfig.ConfigType... cconfigs) Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration and enable a set of boolean compiler configurations.- Parameters:
- dmlconfig- a dml configuration.
- cconfigs- one or many boolean compiler configurations to enable.
 
 - 
Connectionpublic Connection(DMLConfig dmlconfig) Connection constructor, the starting point for any other JMLC API calls. This variant allows to pass a global dml configuration.- Parameters:
- dmlconfig- a dml configuration.
 
 
- 
 - 
Method Detail- 
setStatisticspublic void setStatistics(boolean stats) Sets a boolean flag indicating if runtime statistics should be gathered Same behavior as in "MLContext.setStatistics()"- Parameters:
- stats- boolean value with true indicating statistics should be gathered
 
 - 
setLineagepublic void setLineage(boolean lt) Sets a boolean flag indicating if lineage trace should be captured- Parameters:
- lt- boolean value with true indicating lineage should be captured
 
 - 
gatherMemStatspublic void gatherMemStats(boolean stats) Sets a boolean flag indicating if memory profiling statistics should be gathered. The option is false by default.- Parameters:
- stats- boolean value with true indicating memory statistics should be gathered
 
 - 
prepareScriptpublic PreparedScript prepareScript(String script, String[] inputs, String[] outputs) Prepares (precompiles) a script and registers input and output variables.- Parameters:
- script- string representing the DML or PyDML script
- inputs- string array of input variables to register
- outputs- string array of output variables to register
- Returns:
- PreparedScript object representing the precompiled script
 
 - 
prepareScriptpublic PreparedScript prepareScript(String script, Map<String,String> args, String[] inputs, String[] outputs) Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.- Parameters:
- script- string representing the DML or PyDML script
- args- map of input parameters ($) and their values
- inputs- string array of input variables to register
- outputs- string array of output variables to register
- Returns:
- PreparedScript object representing the precompiled script
 
 - 
prepareScriptpublic PreparedScript prepareScript(String script, Map<String,String> nsscripts, Map<String,String> args, String[] inputs, String[] outputs) Prepares (precompiles) a script, sets input parameter values, and registers input and output variables.- Parameters:
- script- string representing of the DML or PyDML script
- nsscripts- map (name, script) of the DML or PyDML namespace scripts
- args- map of input parameters ($) and their values
- inputs- string array of input variables to register
- outputs- string array of output variables to register
- Returns:
- PreparedScript object representing the precompiled script
 
 - 
closepublic void close() Close connection to SystemDS, which clears the thread-local DML and compiler configurations.- Specified by:
- closein interface- AutoCloseable
- Specified by:
- closein interface- Closeable
 
 - 
readScriptpublic String readScript(String fname) throws IOException Read a DML or PyDML file as a string.- Parameters:
- fname- the filename of the script
- Returns:
- string content of the script file
- Throws:
- IOException- if IOException occurs
 
 - 
readDoubleMatrixpublic double[][] readDoubleMatrix(String fname) throws IOException Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.- Parameters:
- fname- the filename of the input matrix
- Returns:
- matrix as a two-dimensional double array
- Throws:
- IOException- if IOException occurs
 
 - 
readDoubleMatrixpublic double[][] readDoubleMatrix(String fname, Types.FileFormat fmt, long rows, long cols, int blen, long nnz) throws IOException Reads an input matrix in arbitrary format from HDFS into a dense double array. NOTE: this call currently only supports default configurations for CSV.- Parameters:
- fname- the filename of the input matrix
- fmt- file format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- blen- block length
- nnz- number of non-zero values, -1 indicates unknown
- Returns:
- matrix as a two-dimensional double array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToDoubleMatrixpublic double[][] convertToDoubleMatrix(String input, String meta) throws IOException Converts an input string representation of a matrix in csv or textcell format into a dense double array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.- Parameters:
- input- string matrix in csv or textcell format
- meta- string representing SystemDS matrix metadata in JSON format
- Returns:
- matrix as a two-dimensional double array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToDoubleMatrixpublic double[][] convertToDoubleMatrix(String input, int rows, int cols) throws IOException Converts an input string representation of a matrix in textcell format into a dense double array.- Parameters:
- input- string matrix in textcell format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- Returns:
- matrix as a two-dimensional double array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToDoubleMatrixpublic double[][] convertToDoubleMatrix(InputStream input, int rows, int cols) throws IOException Converts an input stream of a string matrix in textcell format into a dense double array.- Parameters:
- input- InputStream to a string matrix in textcell format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- Returns:
- matrix as a two-dimensional double array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToDoubleMatrixpublic double[][] convertToDoubleMatrix(InputStream input, int rows, int cols, String format) throws IOException Converts an input stream of a string matrix in csv or textcell format into a dense double array.- Parameters:
- input- InputStream to a string matrix in csv or textcell format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- format- input format of the given stream
- Returns:
- matrix as a two-dimensional double array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToMatrixpublic MatrixBlock convertToMatrix(String input, String meta) throws IOException Converts an input string representation of a matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.- Parameters:
- input- string matrix in csv or textcell format
- meta- string representing SystemDS matrix metadata in JSON format
- Returns:
- matrix as a matrix block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToMatrixpublic MatrixBlock convertToMatrix(InputStream input, String meta) throws IOException Converts an input stream of a string matrix in csv or textcell format into a matrix block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.- Parameters:
- input- InputStream to a string matrix in csv or textcell format
- meta- string representing SystemDS matrix metadata in JSON format
- Returns:
- matrix as a matrix block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToMatrixpublic MatrixBlock convertToMatrix(String input, int rows, int cols) throws IOException Converts an input string representation of a matrix in textcell format into a matrix block.- Parameters:
- input- string matrix in textcell format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- Returns:
- matrix as a matrix block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToMatrixpublic MatrixBlock convertToMatrix(InputStream input, int rows, int cols) throws IOException Converts an input stream of a string matrix in text format into a matrix block.- Parameters:
- input- InputStream to a string matrix in text format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- Returns:
- matrix as a matrix block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToMatrixpublic MatrixBlock convertToMatrix(InputStream input, int rows, int cols, String format) throws IOException Converts an input stream of a string matrix in csv or text format into a matrix block.- Parameters:
- input- InputStream to a string matrix in csv or text format
- rows- number of rows in the matrix
- cols- number of columns in the matrix
- format- input format of the given stream
- Returns:
- matrix as a matrix block
- Throws:
- IOException- if IOException occurs
 
 - 
readStringFramepublic String[][] readStringFrame(String fname) throws IOException Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.- Parameters:
- fname- the filename of the input frame
- Returns:
- frame as a two-dimensional string array
- Throws:
- IOException- if IOException occurs
 
 - 
readStringFramepublic String[][] readStringFrame(String fname, Types.FileFormat fmt, long rows, long cols) throws IOException Reads an input frame in arbitrary format from HDFS into a dense string array. NOTE: this call currently only supports default configurations for CSV.- Parameters:
- fname- the filename of the input frame
- fmt- file format type
- rows- number of rows in the frame
- cols- number of columns in the frame
- Returns:
- frame as a two-dimensional string array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToStringFramepublic String[][] convertToStringFrame(String input, String meta) throws IOException Converts an input string representation of a frame in csv or textcell format into a dense string array. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.- Parameters:
- input- string frame in csv or textcell format
- meta- string representing SystemDS frame metadata in JSON format
- Returns:
- frame as a two-dimensional string array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToStringFramepublic String[][] convertToStringFrame(String input, int rows, int cols) throws IOException Converts an input stream of a string frame in textcell format into a dense string array.- Parameters:
- input- string frame in textcell format
- rows- number of rows in the frame
- cols- number of columns in the frame
- Returns:
- frame as a two-dimensional string array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToStringFramepublic String[][] convertToStringFrame(InputStream input, int rows, int cols) throws IOException Converts an input stream of a string frame in textcell format into a dense string array.- Parameters:
- input- InputStream to a string frame in text format
- rows- number of rows in the frame
- cols- number of columns in the frame
- Returns:
- frame as a two-dimensional string array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToStringFramepublic String[][] convertToStringFrame(InputStream input, int rows, int cols, String format) throws IOException Converts an input stream of a string frame in csv or text format into a dense string array.- Parameters:
- input- InputStream to a string frame in csv or text format
- rows- number of rows in the frame
- cols- number of columns in the frame
- format- input format of the given stream
- Returns:
- frame as a two-dimensional string array
- Throws:
- IOException- if IOException occurs
 
 - 
convertToFramepublic FrameBlock convertToFrame(String input, String meta) throws IOException Converts an input string representation of a frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.- Parameters:
- input- string frame in csv or textcell format
- meta- string representing SystemDS frame metadata in JSON format
- Returns:
- frame as a frame block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToFramepublic FrameBlock convertToFrame(InputStream input, String meta) throws IOException Converts an input stream of a string frame in csv or textcell format into a frame block. The meta data string is the SystemDS generated .mtd file including the number of rows and columns.- Parameters:
- input- InputStream to a string frame in csv or textcell format
- meta- string representing SystemDS frame metadata in JSON format
- Returns:
- frame as a frame block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToFramepublic FrameBlock convertToFrame(String input, int rows, int cols) throws IOException Converts an input string representation of a frame in textcell format into a frame block.- Parameters:
- input- string frame in textcell format
- rows- number of rows in the frame
- cols- number of columns in the frame
- Returns:
- frame as a frame block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToFramepublic FrameBlock convertToFrame(InputStream input, int rows, int cols) throws IOException Converts an input stream of a string frame in textcell format into a frame block.- Parameters:
- input- InputStream to a string frame in textcell format
- rows- number of rows in the frame
- cols- number of columns in the frame
- Returns:
- frame as a frame block
- Throws:
- IOException- if IOException occurs
 
 - 
convertToFramepublic FrameBlock convertToFrame(InputStream input, int rows, int cols, String format) throws IOException Converts an input stream of a frame in csv or textcell format into a frame block.- Parameters:
- input- InputStream to a string frame in csv or textcell format
- rows- number of rows in the frame
- cols- number of columns in the frame
- format- input format of the given stream
- Returns:
- frame as a frame block
- Throws:
- IOException- if IOException occurs
 
 - 
readTransformMetaDataFromFilepublic FrameBlock readTransformMetaDataFromFile(String metapath) throws IOException Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.- Parameters:
- metapath- hdfs file path to meta data directory
- Returns:
- FrameBlock object representing transform metadata
- Throws:
- IOException- if IOException occurs
 
 - 
readTransformMetaDataFromFilepublic FrameBlock readTransformMetaDataFromFile(String spec, String metapath) throws IOException Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.- Parameters:
- spec- transform specification as json string
- metapath- hdfs file path to meta data directory
- Returns:
- FrameBlock object representing transform metadata
- Throws:
- IOException- if IOException occurs
 
 - 
readTransformMetaDataFromFilepublic FrameBlock readTransformMetaDataFromFile(String spec, String metapath, String colDelim) throws IOException Reads transform meta data from an HDFS file path and converts it into an in-memory FrameBlock object.- Parameters:
- spec- transform specification as json string
- metapath- hdfs file path to meta data directory
- colDelim- separator for processing column names in the meta data file 'column.names'
- Returns:
- FrameBlock object representing transform metadata
- Throws:
- IOException- if IOException occurs
 
 - 
readTransformMetaDataFromPathpublic FrameBlock readTransformMetaDataFromPath(String metapath) throws IOException Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.- Parameters:
- metapath- resource path to meta data directory
- Returns:
- FrameBlock object representing transform metadata
- Throws:
- IOException- if IOException occurs
 
 - 
readTransformMetaDataFromPathpublic FrameBlock readTransformMetaDataFromPath(String spec, String metapath) throws IOException Reads transform meta data from the class path and converts it into an in-memory FrameBlock object. The column names in the meta data file 'column.names' is processed with default separator ','.- Parameters:
- spec- transform specification as json string
- metapath- resource path to meta data directory
- Returns:
- FrameBlock object representing transform metadata
- Throws:
- IOException- if IOException occurs
 
 - 
readTransformMetaDataFromPathpublic FrameBlock readTransformMetaDataFromPath(String spec, String metapath, String colDelim) throws IOException Reads transform meta data from the class path and converts it into an in-memory FrameBlock object.- Parameters:
- spec- transform specification as json string
- metapath- resource path to meta data directory
- colDelim- separator for processing column names in the meta data file 'column.names'
- Returns:
- FrameBlock object representing transform metadata
- Throws:
- IOException- if IOException occurs
 
 
- 
 
-