snudda.detect.prune

class snudda.detect.prune.SnuddaPrune(network_path, logfile=None, logfile_name=None, rc=None, d_view=None, role='master', verbose=False, config_file=None, scratch_path=None, h5libver='latest', random_seed=None, keep_files=False, all_neuron_pair_synapses_share_parameter_id=True)

Prunes detected network synapses to match pairwise connectivity statistics.

Constructor.

Parameters

• network_path (str) – Network directory

• logfile – File pointer to logfile, derived from logfile_name if not given

• logfile_name (str) – Path to logfile, if given will set logfile

• rc – ipyparallel RemoteClient, used for parallel execution

• d_view – ipyparallel direct view, derived from rc if not given

• role (str, optional) – “master” or “worker”

• verbose (bool) – Verbose output, default False

• config_file (str, optional) – Path to config file, default is to use same config file as for detect, but the user can override it. Useful when testing different pruning values in combination with keep_files=True.

• scratch_path (str, optional) – Path to scratch directory for temporary files

• h5libver (str, optional) – Default “latest”

• random_seed (int, optinoal) – Random seed for pruning

• keep_files (bool, optional) – If True then you can redo pruning multiple times without reruning detect

• all_neuron_pair_synapses_share_parameter_id (bool) – Instead of each synapse having a unique parameter_id all synapses between the same neuron pair will have the same parameter id.

big_merge_helper(neuron_range, merge_data_type)

Gathers synapses (or gap junctions) belonging to neuron_range from multiple hyper voxels into one merge file.

Parameters

• neuron_range – Range of neurons

• merge_data_type – “synapses” or “gapJunctions”

buffer_merge_write(synapse_matrix_loc, synapses=None, flush=False)

Writes buffered synapses to file.

Parameters

• synapse_matrix_loc (str) – location of synapses in HDF5 file

• synapses – Synapse to write

• flush (bool) – Buffer needs to be flushed at last call, otherwise all is not written to file

check_hyper_voxel_integrity(hypervoxel_file, hypervoxel_file_name, verbose=False)

Sanity check on hyper voxels. Checks size of voxels, hyper voxels, simulation origo, hyper voxel origo, config files, position files, SlurmID and axon stump flags match.

Parameters

• hypervoxel_file – HDF5 file object

• hypervoxel_file_name – Name of HDF5 file

• verbose (bool) – Print extra information, default False

check_network_config_integrity(config_file)

This checks that all connections included in the pruning, were present in the detection. If some are missing in the detection phase, then they would incorrectly be missing after pruning.

Parameters

config_file (str) – Path to new config file

clean_up_merge_files()

Cleans up merge files.

clean_up_merge_read_buffers()

Housekeeping, clean up merge read buffers.

cleanup()

Removes files in temp and voxels directories, freeing up diskspace.

combine_files(source_filenames, merge_data_type, output_filename=None)

Combines synapse files after pruning in network-synapses.hdf5.

Parameters

• source_filenames (list) – List with path to source files

• merge_data_type (str) – “synapses” or “gapJunctions”

• output_filename (str) – Output file name

static complete_pruning_info(prune_info)

This makes sure all the variables exist, that way prune_synapses_helper does not have to check, but can assume that they will exist

Parameters

prune_info (dict) – Dictionary with pruning information to be complemented.

file_row_lookup_iterator_subset(h5mat_lookup, min_dest_id, max_dest_id, chunk_size=10000)

File row lookup iterator, working on a subset of the synapse matrix. min_dest_id and max_dest_id are inclusive, only synapses with dest_id in that range are iterated over

Parameters

• h5mat_lookup – Synapse lookup

• min_dest_id – Minimum neuron destination ID for synapses (inclusive)

• max_dest_id – Maximum neuron destination ID for synapses (inclusive)

• chunk_size – Chunk size for processing

find_ranges(synapses, num_workers, start_pos=0, num_syn=None)

Find which ranges of the synapse matrix that each worker should take care of

Parameters

• synapses – Synapse matrix

• num_workers (int) – Number of workers

• start_pos (int) – Starting from position

• num_syn (int) – Number of synapses

gather_neuron_synapses()

Goes through the hyper voxel synapse files, extracts a range of neurons, and puts their synapses in its own file. Parallel execution, all neurons.

get_hyper_voxel_list(neuron_range)

Gather a list of all hyper voxels which contains synapses belonging to neurons in neuron_range.

Parameters

neuron_range – Range of neurons

get_merge_info()

Reads merge info written by save_merge_info.

Returns

merge_files_syn : List of files containing synapses merge_neuron_range_syn : List of neuron ranges in each file merge_syn_ctr : List of synapse counts for each file merge_files_gj : List of files containing gap junctions merge_neuron_range_gj : List of neuron ranges in each file merge_gj_ctr : List of gap junction count for each file

Return type

tuple of data

get_merge_info_helper()

Helper function for get_merge_info. Reads merge info written by save_merge_info. Please use get_merge_info, which also includes sanity checks on the data returned.

Returns

merge_files_syn : List of files containing synapses merge_neuron_range_syn : List of neuron ranges in each file merge_syn_ctr : List of synapse counts for each file merge_files_gj : List of files containing gap junctions merge_neuron_range_gj : List of neuron ranges in each file merge_gj_ctr : List of gap junction count for each file

Return type

tuple of data

get_neuron_random_seeds()

Derive neuron random seeds from pruning master seed.

load_pruning_information(config_file=None)

Parse the connection information in the config file

Parameters

config_file (str) – Path to config file

make_type_numbering()

Normally we use type names as lookups in connection_distribution, but since we will do this many millions of times, we create an temporary type_id number.

open_hyper_voxel(hyper_voxel_id, verbose=False, verify=True)

Helper function, opens hyper voxel.

Parameters

• hyper_voxel_id (int) – ID of hyper voxel to open

• verbose (bool) – Print extra information

• verify (bool) – Verify hypervoxel integrity, default=True

open_work_history_file(work_history_file=None, config_file=None)

Opens work history file.

Parameters

• work_history_file (str, optional) – Path to work history file (if you want to override default, network-detect-worklog.hdf5)

• config_file (str, optional) – Path to config file, if you want to override file used for detection.

prune()

Merges output files from detect.py and prunes the synapses. Writes network-synapses.hdf5

prune_synapses(synapse_file, output_filename, merge_data_type, row_range=None, close_input_file=True, close_out_file=True)

Prune synapses.

Parameters

• synapse_file – Path to file with putative synapses

• output_filename – Path to file to write pruned synapses to

• merge_data_type – “synapses” or “gapJunctions”

• row_range – Synapse row range to prune

• close_input_file (bool) – Close input files after

• close_out_file (bool) – Close output file after

prune_synapses_helper(synapses, output_file, merge_data_type)

Helper function to prunes synapses. It takes a subset of the synapse matrix as input, as it needs to keep all synapses onto the same neuron in memory at the same time. It buffers writes until the end.

Parameters

• synapses – subset of synapse matrix that fits in memory

• output_file – where to write synapses, assumed to already exist

• merge_data_type – “synapses” or “gapJunctions”

prune_synapses_parallel(synapse_file, synapse_ctr, merge_data_type='synapses', close_input_file=True)

Helper function, performs synapse (or gap junction) pruning in parallel.

Parameters

• synapse_file (list) – List with paths to synapse files

• synapse_ctr (int) – List or array with number of synapses

• merge_data_type (str) – “synapses” or “gapJunctions”

• close_input_file (bool) – Close input file after, or keep open (default True)

save_merge_info(merge_files_syn, merge_neuron_range_syn, merge_syn_ctr, merge_files_gj, merge_neuron_range_gj, merge_gj_ctr)

Writes merge info to file (pruning_merge_info.json), so prune can be rerun without having to re-merge detection hyper voxel files.

Parameters

• merge_files_syn – List of files containing synapses

• merge_neuron_range_syn – List of neuron ranges in each file

• merge_syn_ctr – List of synapse counts for each file

• merge_files_gj – List of files containing gap junctions

• merge_neuron_range_gj – List of neuron ranges in each file

• merge_gj_ctr – List of gap junction count for each file

set_scratch_path(scratch_path=None)

Sets scratch path. OBS, need to call open_work_history_file first.

Parameters

scratch_path (str) – Path to scratch directory

setup_merge_file(big_cache=False, outfile_name=None, save_morphologies=False, num_synapses=None, num_gap_junctions=None, delete_after=True)

Sets up merge file, which is done after detection, but before synapse pruning. Writes network-putative-synapses-MERGED.hdf5

Parameters

• big_cache (bool) – Increase HDF5 cache size

• outfile_name (str) – Name of output file name (default network-putative-synapses-MERGED.hdf5)

• num_synapses (int) – Number of expected synapses

• num_gap_junctions (int) – Number of expected gap junctions

• delete_after (bool) – Clean up hypervoxel files after

setup_output_file(output_file=None, save_morphologies=False)

Sets up output file (network-synapses.hdf5), copying over meta data.

Parameters

output_file (str, optional) – Path to output file, default network-synapses.hdf5

setup_parallel(d_view)

Setup workers for parallel execution.

Parameters

d_view – ipyparallel direct view object

static synapse_set_iterator(h5mat_lookup, h5mat, chunk_size=10000, lookup_iterator=None)

Iterator over synapses.

Parameters

• h5mat_lookup – Synapse lookup table

• h5mat – Synapse matrix

• chunk_size – Chunk size to process

• lookup_iterator – Synapse lookup iterator

write_log(text, flush=True, is_error=False, force_print=False)

Writes to log file. Use setup_log first. Text is only written to screen if self.verbose=True, or is_error = True, or force_print = True.

test (str) : Text to write flush (bool) : Should all writes be flushed to disk directly? is_error (bool) : Is this an error, always written. force_print (bool) : Force printing, even if self.verbose=False.