Simulation control

Trajectory

class l2l.utils.trajectory.Trajectory(**keyword_args)

Bases: object

The trajectory is a class which holds the history of the parameter space exploration, defines the current parameters to be explored and holds the results from each execution. Based on the pypet trajectory concept: https://github.com/SmokinCaterpillar/pypet

f_add_parameter_group(name, comment='')

Adds a new parameter group :param name: name of the new parameter group :param comment: ignored for the moment. Kept to match pypet interface.

f_add_parameter_to_group(group_name, key, val)

Adds a parameter to an already existing group.

Parameters:

• group_name – Name of the group where the parameter should be added

• key – Name of the parameter to be added

• val – Value of the parameter

Throws an exception if the group does not exist

f_add_result(key, val, comment='')

Adds a result to the trajectory :param key: it identifies either a generation params result group or another result :param val: The value to be added to the results TODO: verify where is the generation_params call performed

f_add_parameter(key, val, comment='')

Adds a parameter to the trajectory :param key: Name of the parameter :param val: Value of the parameter :param comment

f_add_derived_parameter(key, val, comment='')

Adds a derived parameter to the trajectory. Match the previous pypet interface. :param key: Name of the parameter :param val: Value of the parameter :param comment:

f_expand(build_dict, fail_safe=True)

The expand function takes care of adding a new generation and individuals to the trajectory This is a critical function to allow the addition of a new generation, called by the optimizer from the postprocessing function :param build_dict: The dictionary containing the new generation id and its individuals :param fail_safe: Currently ignored

Individual

class l2l.utils.individual.Individual(generation=0, ind_idx=0, params=[])

Bases: ParameterGroup

This class represents individuals in the parameter search. It derives from a Parameter group. The main elements which make an individual are the ID of its generation, its individual ID and the params specific for its run.

ParamterGroup

class l2l.utils.groups.ParameterGroup

Bases: object

This class is a Dictionary which can be used to store parameters. It is used to fit the pypet already existing interface with the trajectory

f_add_parameter(key, val, comment='')

Adds parameter with name key and value val. The comment is ignored for the moment but kept for compatibility with the pypet groups :param key: Name of the parameter :param val: Value of the parameter :param comment: Ignores for the moment

Environment

class l2l.utils.environment.Environment(*args, **keyword_args)

Bases: object

The Environment class takes the place of the pypet Environment and provides the required functionality to execute the inner loop. Based on the pypet environment concept: https://github.com/SmokinCaterpillar/pypet

run()

Runs all generations of the optimizees using the runner.

add_postprocessing(func)

Function to add a postprocessing step :param func: the function which performs the postprocessing. Postprocessing is the step where the results are assessed in order to produce a new set of parameters for the next generation.

enable_logging()

Function to enable logging TODO think about removing this.

disable_logging()

Function to enable logging

Experiment

class l2l.utils.experiment.Experiment(root_dir_path)

Bases: object

prepare_experiment(**kwargs)

Prepare the experiment by creating the enviroment and :param kwargs: optional dictionary, contains

• name: str, name of the run, Default: L2L-run

• trajectory_name: str, name of the trajectory, Default: trajectory

• checkpoint: object, trajectory object

• log_stdout: bool, if stdout should be sent to logs, Default:False

• runner_params: dict, User specified parameters for the runner.

  See notes section for default runner parameters

• multiprocessing, bool, enable multiprocessing, Default: False

• debug, bool, enable verbose mode to get more detailed logs for debugging,

  Default: False

• stop_run, bool, when an error occures the execution is stoped, Default: False

• timeout, bool, stops execution after 2 hours if it is not finished by then,

  Default: True

-overwrite, bool, specifies whether existing files should be overwritten

Default: False

:return traj, trajectory object :return all_runner_params, dict, a dictionary with all parameters for the runner

given by the user and default ones

:notes

Default runner parameters are:

• srun: “”

• exec: ‘python3 + “self.paths.simulation_path/run_optimizee.py”’

• max_workers: 32

• work_path: self.paths.root_dir_path,

• paths_obj: self.paths

run_experiment(optimizer, optimizee, optimizer_parameters=None, optimizee_parameters=None)

Runs the simulation with all parameter combinations

Optimizee and optimizer object are required as well as their parameters as namedtuples.

Parameters:

• optimizee – optimizee object

• optimizee_parameters – Namedtuple, optional, parameters of the optimizee

• optimizer – optimizer object

• optimizer_parameters – Namedtuple, optional, parameters of the optimizer

end_experiment(optimizer)

Ends the experiment and disables the logging

Parameters:

optimizer – optimizer object

:return traj, trajectory object :return path, Path object

load_trajectory(traj_path)

Loads a trajectory from a given file :param traj_path: path to the trajectory file :return traj: trajectory object

Runner

class l2l.utils.runner.Runner(trajectory, iterations)

Bases: object

A class used to launch the individual optimizees in parallel within the available computing resources. Takes care of launching, monitoring and ending the execution of the individuals. It generates workers which are assigned one or more individuals to be executed within each generation. It also takes care of collecting the results and relaunching individuals if there are runtime or logic errors associated with the execution.

…

Methods

collect_results_from_run(self, generation, individuals) run(self, trajectory, generation) produce_run_command(idx) launch(idx) launch_workers() close_workers() restart_worker(w_id) simulate_generation(gen, n_inds) prepare_run_file() dump_traj(trajectory) create_zipfile(folder, filename)

collect_results_from_run(generation, individuals)

Collects the results generated by each individual in the generation. Results are, for the moment, stored in individual binary files. :param generation: generation id :param individuals: list of individuals which were executed in this generation :return results: a list containing objects produced as results of the execution of each individual

run(trajectory, generation)

Takes care of running the generation by executing run_optimizee.py in parallel, waiting for the execution and gathering the results. :param trajectory: trajectory object storing individual parameters for each generation :param generation: id of the generation :return results: a list containing objects produced as results of the execution of each individual

produce_run_command(idx)

Generates a string that can be used to launch an instance of the optimizee with specific parameters, also called an individual. :param idx: the id of the individual to be launched.

launch_worker(w_id)

This function uses the subprocess.Popen function to launch the command required to initialize a parallel worker. This worker will stay alive during the whole optimization run and will receive the id of the individuals it will execute each generation. It also generates the pipes (files in a shared file system) to communicate between the runner and the worker. Each worker has its own file in the path ‘individual_logs’ with the worker_ prefix. The logs pertaining to each individual are also stored in ‘individual_logs’ with err_ and out_ prefixes. :param idx: the id of the individual to be launched.

launch_workers()

Takes care of launching enough workers as allowed by the available computing resources.

close_workers()

Makes sure all workers are notified that the optimization run is over and closes all open pipes and files.

restart_worker(w_id)

Takes care of handling the restart of a worker and its associated individual which failed by any reason, either runtime or logic. :param w_id: the id of the worker to be launched.

restart_individual(gen, idx)

Takes care of handling the restart of an individual which failed by any reason, either runtime or logic. :param gen: the current generation :param idx: the id of the individual to be launched.i

populate_free_workers(gen)

simulate_generation(gen, n_inds)

Executes n_inds srun commands, waits for them to finish and writes their exit codes to ‘exit_codes.log’ :param gen: the current generation :param n_inds: the number of individuals within the generation

prepare_run_file()

Writes a python run file which takes care of loading the optimizee from a binary file, the trajectory object of each individual. Then executes the ‘simulate’ function of the optimizee using the trajectory and writes the results in a binary file. :param path_ready: path to store the ready files :return true if all files are present, false otherwise

dump_traj(trajectory)

Dumpes trajectory files. :param trajectory, object to be dumped

create_zipfile(folder, filename)

Creates zipfile and deletes files included in the zip file :param folder: path to folder containing the files :param filename: filename of the created zip file