dataconverters

Pre-configured combinations of writers and readers.

class graphnet.data.pre_configured.dataconverters.I3ToParquetConverter(gcd_rescue, extractors, outdir, index_column, num_workers, i3_filters)[source]

Bases: DataConverter

Preconfigured DataConverter for converting i3 files to parquet files.

Convert I3 files to Parquet.

Parameters:

  • gcd_rescue (str) – gcd_rescue: Path to a GCD file that will be used if no GCD file is found in subfolder. I3Reader will recursively search the input directory for I3-GCD file pairs. By IceCube convention, a folder containing i3 files will have an accompanying GCD file. However, in some cases, this convention is broken. In cases where a folder contains i3 files but no GCD file, the gcd_rescue is used instead.

  • extractors (List[I3Extractor]) – The `Extractor`(s) that will be applied to the input files.

  • outdir (str) – The directory to save the files in.

  • icetray_verbose – Set the level of verbosity of icetray. Defaults to 0.

  • index_column (str, default: 'event_no') – Name of the event id column added to the events. Defaults to “event_no”.

  • num_workers (int, default: 1) – The number of CPUs used for parallel processing. Defaults to 1 (no multiprocessing).

  • i3_filters (Union[I3Filter, List[I3Filter]], default: None) – Instances of I3Filter to filter PFrames. Defaults to NullSplitI3Filter.

class graphnet.data.pre_configured.dataconverters.I3ToSQLiteConverter(gcd_rescue, extractors, outdir, index_column, num_workers, i3_filters, max_table_size)[source]

Bases: DataConverter

Preconfigured DataConverter for converting i3 files to SQLite files.

Convert I3 files to SQLite.

Parameters:

  • gcd_rescue (str) – gcd_rescue: Path to a GCD file that will be used if no GCD file is found in subfolder. I3Reader will recursively search the input directory for I3-GCD file pairs. By IceCube convention, a folder containing i3 files will have an accompanying GCD file. However, in some cases, this convention is broken. In cases where a folder contains i3 files but no GCD file, the gcd_rescue is used instead.

  • extractors (List[I3Extractor]) – The `Extractor`(s) that will be applied to the input files.

  • outdir (str) – The directory to save the files in.

  • icetray_verbose – Set the level of verbosity of icetray. Defaults to 0.

  • index_column (str, default: 'event_no') – Name of the event id column added to the events. Defaults to “event_no”.

  • num_workers (int, default: 1) – The number of CPUs used for parallel processing. Defaults to 1 (no multiprocessing).

  • i3_filters (Union[I3Filter, List[I3Filter]], default: None) – Instances of I3Filter to filter PFrames. Defaults to NullSplitI3Filter.

  • max_table_size (Optional[int], default: None) – Maximum size of the SQLite tables. Default None.

class graphnet.data.pre_configured.dataconverters.ParquetToSQLiteConverter(extractors, outdir, index_column, num_workers)[source]

Bases: DataConverter

Preconfigured DataConverter for converting Parquet to SQLite files.

This class converts Parquet files written by ParquetWriter to SQLite.

Convert internal Parquet files to SQLite.

Parameters:

  • extractors (List[ParquetExtractor]) – The `Extractor`(s) that will be applied to the input files.

  • outdir (str) – The directory to save the files in.

  • icetray_verbose – Set the level of verbosity of icetray. Defaults to 0.

  • index_column (str, default: 'event_no') – Name of the event id column added to the events. Defaults to “event_no”.

  • num_workers (int, default: 1) – The number of CPUs used for parallel processing. Defaults to 1 (no multiprocessing).