Package org.apache.sysds.hops.recompile
Class Recompiler
- java.lang.Object
- 
- org.apache.sysds.hops.recompile.Recompiler
 
- 
 public class Recompiler extends Object Dynamic recompilation of hop dags to runtime instructions, which includes the following substeps: (1) deep copy hop dag, (2) refresh matrix characteristics, (3) apply dynamic rewrites, (4) refresh memory estimates, (5) construct lops (incl operator selection), and (6) generate runtime program (incl piggybacking).
- 
- 
Nested Class SummaryNested Classes Modifier and Type Class Description static classRecompiler.ResetType
 - 
Constructor SummaryConstructors Constructor Description Recompiler()
 - 
Method SummaryAll Methods Static Methods Concrete Methods Modifier and Type Method Description static booleancheckCPCheckpoint(DataCharacteristics dc)static booleancheckCPReblock(ExecutionContext ec, String varin)CP Reblock check for spark instructions; in contrast to MR, we can not rely on the input file sizes because inputs might be passed via rdds.static ArrayList<Hop>deepCopyHopsDag(List<Hop> hops)Deep copy of hops dags for parallel recompilation.static HopdeepCopyHopsDag(Hop hops)Deep copy of hops dags for parallel recompilation.static voidexecuteInMemoryReblock(ExecutionContext ec, String varin, String varout)static voidexecuteInMemoryReblock(ExecutionContext ec, String varin, String varout, LineageItem litem)static voidextractDAGOutputStatistics(ArrayList<Hop> hops, LocalVariableMap vars)static voidextractDAGOutputStatistics(ArrayList<Hop> hops, LocalVariableMap vars, boolean overwrite)static voidextractDAGOutputStatistics(Hop hop, LocalVariableMap vars, boolean overwrite)static voidrClearLops(Hop hop)Clearing lops for a given hops includes to (1) remove the reference to constructed lops and (2) clear the exec type (for consistency).static ArrayList<Instruction>recompileHopsDag(Hop hop, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)static ArrayList<Instruction>recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, ExecutionContext ec, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)static ArrayList<Instruction>recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid)static ArrayList<Instruction>recompileHopsDag2Forced(Hop hop, long tid, Types.ExecType et)static ArrayList<Instruction>recompileHopsDag2Forced(StatementBlock sb, ArrayList<Hop> hops, long tid, Types.ExecType et)static ArrayList<Instruction>recompileHopsDagInstructions(Hop hop)static ArrayList<Instruction>recompileHopsDagInstructions(StatementBlock sb, ArrayList<Hop> hops)static voidrecompileProgramBlockHierarchy(ArrayList<ProgramBlock> pbs, LocalVariableMap vars, long tid, boolean inplace, Recompiler.ResetType resetRecompile)static voidrecompileProgramBlockHierarchy2Forced(ArrayList<ProgramBlock> pbs, long tid, HashSet<String> fnStack, Types.ExecType et)Method to recompile program block hierarchy to forced execution time.static voidrecompileProgramBlockInstructions(ProgramBlock pb)This method does NO full program block recompile (no stats update, no rewrites, no recursion) but only regenerates lops and instructions.static RecompileStatusreconcileUpdatedCallVarsIf(RecompileStatus oldStatus, RecompileStatus callStatusIf, RecompileStatus callStatusElse, StatementBlock sb)static LocalVariableMapreconcileUpdatedCallVarsIf(LocalVariableMap oldCallVars, LocalVariableMap callVarsIf, LocalVariableMap callVarsElse, StatementBlock sb)static booleanreconcileUpdatedCallVarsLoops(RecompileStatus oldCallStatus, RecompileStatus callStatus, StatementBlock sb)static booleanreconcileUpdatedCallVarsLoops(LocalVariableMap oldCallVars, LocalVariableMap callVars, StatementBlock sb)static voidreinitRecompiler()Re-initializes the recompiler according to the current optimizer flags.static voidremoveUpdatedScalars(LocalVariableMap callVars, StatementBlock sb)Remove any scalar variables from the variable map if the variable is updated in this block.static booleanrequiresRecompilation(ArrayList<Hop> hops)static booleanrequiresRecompilation(Hop hop)static intrGetMaxParallelism(List<Hop> hops)static intrGetMaxParallelism(Hop hop)static voidrRecompileProgramBlock2Forced(ProgramBlock pb, long tid, HashSet<String> fnStack, Types.ExecType et)static voidrReplaceLiterals(Hop hop, ExecutionContext ec, boolean scalarsOnly)public interface to package local literal replacementstatic voidrReplaceLiterals(Hop hop, LocalVariableMap vars, boolean scalarsOnly)static voidrSetExecType(Hop hop, Types.ExecType etype)static voidrSetMaxParallelism(List<Hop> hops, int k)static voidrSetMaxParallelism(Hop hop, int k)static voidrUpdateFunctionNames(Hop hop, long pid)static voidrUpdateStatistics(Hop hop, LocalVariableMap vars)static voidupdateFunctionNames(ArrayList<Hop> hops, long pid)
 
- 
- 
- 
Method Detail- 
reinitRecompilerpublic static void reinitRecompiler() Re-initializes the recompiler according to the current optimizer flags.
 - 
recompileHopsDagpublic static ArrayList<Instruction> recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, ExecutionContext ec, RecompileStatus status, boolean inplace, boolean replaceLit, long tid) 
 - 
recompileHopsDagpublic static ArrayList<Instruction> recompileHopsDag(StatementBlock sb, ArrayList<Hop> hops, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid) 
 - 
recompileHopsDagpublic static ArrayList<Instruction> recompileHopsDag(Hop hop, LocalVariableMap vars, RecompileStatus status, boolean inplace, boolean replaceLit, long tid) 
 - 
recompileHopsDag2Forcedpublic static ArrayList<Instruction> recompileHopsDag2Forced(StatementBlock sb, ArrayList<Hop> hops, long tid, Types.ExecType et) 
 - 
recompileHopsDag2Forcedpublic static ArrayList<Instruction> recompileHopsDag2Forced(Hop hop, long tid, Types.ExecType et) 
 - 
recompileHopsDagInstructionspublic static ArrayList<Instruction> recompileHopsDagInstructions(StatementBlock sb, ArrayList<Hop> hops) 
 - 
recompileHopsDagInstructionspublic static ArrayList<Instruction> recompileHopsDagInstructions(Hop hop) 
 - 
recompileProgramBlockHierarchypublic static void recompileProgramBlockHierarchy(ArrayList<ProgramBlock> pbs, LocalVariableMap vars, long tid, boolean inplace, Recompiler.ResetType resetRecompile) 
 - 
recompileProgramBlockHierarchy2Forcedpublic static void recompileProgramBlockHierarchy2Forced(ArrayList<ProgramBlock> pbs, long tid, HashSet<String> fnStack, Types.ExecType et) Method to recompile program block hierarchy to forced execution time. This affects also referenced functions and chains of functions. Use et==null in order to release the forced exec type.- Parameters:
- pbs- list of program blocks
- tid- thread id
- fnStack- function stack
- et- execution type
 
 - 
recompileProgramBlockInstructionspublic static void recompileProgramBlockInstructions(ProgramBlock pb) throws IOException This method does NO full program block recompile (no stats update, no rewrites, no recursion) but only regenerates lops and instructions. The primary use case is recompilation after are hop configuration changes which allows to preserve statistics (e.g., propagated worst case stats from other program blocks) and better performance for recompiling individual program blocks.- Parameters:
- pb- program block
- Throws:
- IOException- if IOException occurs
 
 - 
requiresRecompilationpublic static boolean requiresRecompilation(Hop hop) 
 - 
deepCopyHopsDagpublic static ArrayList<Hop> deepCopyHopsDag(List<Hop> hops) Deep copy of hops dags for parallel recompilation.- Parameters:
- hops- list of high-level operators
- Returns:
- list of high-level operators
 
 - 
deepCopyHopsDagpublic static Hop deepCopyHopsDag(Hop hops) Deep copy of hops dags for parallel recompilation.- Parameters:
- hops- high-level operator
- Returns:
- high-level operator
 
 - 
rUpdateFunctionNamespublic static void rUpdateFunctionNames(Hop hop, long pid) 
 - 
reconcileUpdatedCallVarsLoopspublic static boolean reconcileUpdatedCallVarsLoops(LocalVariableMap oldCallVars, LocalVariableMap callVars, StatementBlock sb) 
 - 
reconcileUpdatedCallVarsLoopspublic static boolean reconcileUpdatedCallVarsLoops(RecompileStatus oldCallStatus, RecompileStatus callStatus, StatementBlock sb) 
 - 
reconcileUpdatedCallVarsIfpublic static LocalVariableMap reconcileUpdatedCallVarsIf(LocalVariableMap oldCallVars, LocalVariableMap callVarsIf, LocalVariableMap callVarsElse, StatementBlock sb) 
 - 
reconcileUpdatedCallVarsIfpublic static RecompileStatus reconcileUpdatedCallVarsIf(RecompileStatus oldStatus, RecompileStatus callStatusIf, RecompileStatus callStatusElse, StatementBlock sb) 
 - 
rRecompileProgramBlock2Forcedpublic static void rRecompileProgramBlock2Forced(ProgramBlock pb, long tid, HashSet<String> fnStack, Types.ExecType et) 
 - 
removeUpdatedScalarspublic static void removeUpdatedScalars(LocalVariableMap callVars, StatementBlock sb) Remove any scalar variables from the variable map if the variable is updated in this block.- Parameters:
- callVars- Map of variables eligible for propagation.
- sb- DML statement block.
 
 - 
extractDAGOutputStatisticspublic static void extractDAGOutputStatistics(ArrayList<Hop> hops, LocalVariableMap vars) 
 - 
extractDAGOutputStatisticspublic static void extractDAGOutputStatistics(ArrayList<Hop> hops, LocalVariableMap vars, boolean overwrite) 
 - 
extractDAGOutputStatisticspublic static void extractDAGOutputStatistics(Hop hop, LocalVariableMap vars, boolean overwrite) 
 - 
rClearLopspublic static void rClearLops(Hop hop) Clearing lops for a given hops includes to (1) remove the reference to constructed lops and (2) clear the exec type (for consistency). The latter is important for advanced optimizers like parfor; otherwise subtle side-effects of program recompilation and hop-lop rewrites possible (e.g., see indexingop hop-lop rewrite in combination parfor rewrite set exec type that eventuelly might lead to unnecessary remote_parfor jobs).- Parameters:
- hop- high-level operator
 
 - 
rUpdateStatisticspublic static void rUpdateStatistics(Hop hop, LocalVariableMap vars) 
 - 
rReplaceLiteralspublic static void rReplaceLiterals(Hop hop, ExecutionContext ec, boolean scalarsOnly) public interface to package local literal replacement- Parameters:
- hop- high-level operator
- ec- Execution context
- scalarsOnly- if true, replace only scalar variables but no matrix operations; if false, apply full literal replacement
 
 - 
rReplaceLiteralspublic static void rReplaceLiterals(Hop hop, LocalVariableMap vars, boolean scalarsOnly) 
 - 
rSetExecTypepublic static void rSetExecType(Hop hop, Types.ExecType etype) 
 - 
rGetMaxParallelismpublic static int rGetMaxParallelism(Hop hop) 
 - 
rSetMaxParallelismpublic static void rSetMaxParallelism(Hop hop, int k) 
 - 
checkCPReblockpublic static boolean checkCPReblock(ExecutionContext ec, String varin) CP Reblock check for spark instructions; in contrast to MR, we can not rely on the input file sizes because inputs might be passed via rdds.- Parameters:
- ec- execution context
- varin- variable
- Returns:
- true if CP reblock?
 
 - 
checkCPCheckpointpublic static boolean checkCPCheckpoint(DataCharacteristics dc) 
 - 
executeInMemoryReblockpublic static void executeInMemoryReblock(ExecutionContext ec, String varin, String varout) 
 - 
executeInMemoryReblockpublic static void executeInMemoryReblock(ExecutionContext ec, String varin, String varout, LineageItem litem) 
 
- 
 
-