pymepix.processing package

Subpackages

• pymepix.processing.logic package
  • Submodules
  • pymepix.processing.logic.centroid_calculator module
  • pymepix.processing.logic.packet_processor module
  • pymepix.processing.logic.processing_step module
  • pymepix.processing.logic.shared_processing_parameter module
  • Module contents

Submodules

pymepix.processing.acquisition module

Module that contains predefined acquisition pipelines for the user to use

class pymepix.processing.acquisition.CentroidPipeline(data_queue, address, longtime)

Bases: pymepix.processing.acquisition.PixelPipeline

A Pixel pipeline that includes centroiding

Same as the pixel pipeline but also includes centroid processing, note that this can be extremely slow when dealing with a huge number of objects

numBlobProcesses

Number of python processes to spawn for centroiding

Setting this will spawn the appropriate number of processes to perform centroiding. Changes take effect on next acquisition.

class pymepix.processing.acquisition.PixelPipeline(data_queue, address, longtime, use_event=False, name='Pixel', event_window=(0, 0.001))

Bases: pymepix.processing.baseacquisition.AcquisitionPipeline

An acquisition pipeline that includes the udpsampler and pixel processor

A pipeline that will read from a UDP address and decode the pixels a useable form. This class can be used as a base for all acqusition pipelines.

pymepix.processing.baseacquisition module

Module deals with managing processing objects to form a data pipeline

class pymepix.processing.baseacquisition.AcquisitionPipeline(name, data_queue)

Bases: pymepix.core.log.Logger

Class that manages various stages

addStage(stage_number, pipeline_klass, *args, num_processes=1, **kwargs)

Adds a stage to the pipeline

getStage(stage_number)

isRunning

stages

start()

Starts all stages

stop()

Stops all stages

class pymepix.processing.baseacquisition.AcquisitionStage(stage, num_processes=1)

Bases: pymepix.core.log.Logger

Defines a single acquisition stage

Usually not created directly. Instead created by AcquisitionPipeline Represent a single pipeline stage and handles management of queues and message passing as well as creation and destruction of processing objects.

Processes are not created until build() is called and do not run until start() is called

Parameters:stage (int) – Initial position in the pipeline, lower stages are executed first

build(input_queue=None, output_queue=None, file_writer=None)

configureStage(pipeline_klass, *args, **kwargs)

Configures the stage with a particular processing class

Parameters:

• pipeline_klass (BasePipeline) – A pipeline class object
• *args – positional arguments to pass into the class init
• **kwargs – keyward arguments to pass into the class init

numProcess

Number of processes to spawn when built

Parameters:value (int) – Number of processes to spawn when acquisition starts Returns:Number of processes Return type:int

outputQueue

processes

setArgs(*args, **kwargs)

stage

Current position in the pipeline

start()

startTrainID()

stop(force=False)

stopTrainID()

pymepix.processing.baseacquisition.main()

pymepix.processing.basepipeline module

Base implementation of objects relating to the processing pipeline

class pymepix.processing.basepipeline.BasePipelineObject(name, input_queue=None, create_output=True, num_outputs=1, shared_output=None, propogate_input=True)

Bases: multiprocessing.context.Process, pymepix.core.log.ProcessLogger

Base class for integration in a processing pipeline

Parameters:

• name (str) – Name used for logging
• input_queue (multiprocessing.Queue, optional) – Data queue to perform work on (usually) from previous step in processing pipeline
• create_output (bool, optional) – Whether this creates its own output queue to pass data, ignored if (Default: True)
• num_outputs (int,optional) – Used with create_output, number of output queues to create (Default: 1)
• shared_output (multiprocessing.Queue, optional) – Data queue to pass results into, useful when multiple processes can put data into the same queue (such as results from centroiding). Ignored if create_output is True (Default: None)
• propogate_input (bool) – Whether the input data should be propgated further down the chain

enable

Enables processing

Determines whether the class will perform processing, this has the result of signalling the process to terminate. If there are objects ahead of it then they will stop receiving data if an input queue is required then it will get from the queue before checking processing This is done to prevent the queue from growing when a process behind it is still working

Parameters:value (bool) – Enable value Returns:Whether the process is enabled or not Return type:bool

classmethod hasOutput()

Defines whether this class can output results or not, e.g. Centroiding can output results but file writing classes do not

Returns:Whether results are generated Return type:bool

outputQueues

Exposes the outputs so they may be connected to the next step

Returns:All of the outputs Return type:list of multiprocessing.Queue

post_run()

Function called after main processing loop, override to

pre_run()

Function called before main processing loop, override to

process(data_type=None, data=None)

Main processing function, override this do perform work

To perform work within the pipeline, a class must override this function. General guidelines include, check for correct data type, and must return None for both if no output is given.

pushOutput(data_type, data)

Pushes results to output queue (if available)

Parameters:

• data_type (int) – Identifier for data type (see MeesageType for types)
• data (any) – Results from processing (must be picklable)

run()

Method to be run in sub-process; can be overridden in sub-class

pymepix.processing.basepipeline.main()

pymepix.processing.datatypes module

Defines data that is passed between processing objects

class pymepix.processing.datatypes.MessageType

Bases: enum.IntEnum

Defines the type of message that is being passed into a multiprocessing queue

CentroidData = 3

Centroided Data

CloseFileCommand = 5

Close File Message

EventData = 2

Event Data

OpenFileCommand = 4

Open File message

PixelData = 1

Decoded Pixel/Trigger Data

RawData = 0

Raw UDP packets

TriggerData = 8

Decoded Triggers

pymepix.processing.pipeline_centroid_calculator module

Processors relating to centroiding

class pymepix.processing.pipeline_centroid_calculator.PipelineCentroidCalculator(centroid_calculator: pymepix.processing.logic.centroid_calculator.CentroidCalculator = <pymepix.processing.logic.centroid_calculator.CentroidCalculator object>, input_queue=None, create_output=True, num_outputs=1, shared_output=None)

Bases: pymepix.processing.basepipeline.BasePipelineObject

Performs centroiding on EventData recieved from Packet processor

process(data_type=None, data=None)

Main processing function, override this do perform work

To perform work within the pipeline, a class must override this function. General guidelines include, check for correct data type, and must return None for both if no output is given.

pymepix.processing.pipeline_packet_processor module

class pymepix.processing.pipeline_packet_processor.PipelinePacketProcessor(packet_processor: pymepix.processing.logic.packet_processor.PacketProcessor = <pymepix.processing.logic.packet_processor.PacketProcessor object>, input_queue=None, create_output=True, num_outputs=1, shared_output=None)

Bases: pymepix.processing.basepipeline.BasePipelineObject

Processes Pixel packets for ToA, ToT, triggers and events

This class, creates a UDP socket connection to SPIDR and recivies the UDP packets from Timepix It then pre-processes them and sends them off for more processing

init_new_process()

post_run()

Function called after main processing loop, override to

pre_run()

Function called before main processing loop, override to

process(data_type=None, data=None)

Main processing function, override this do perform work

To perform work within the pipeline, a class must override this function. General guidelines include, check for correct data type, and must return None for both if no output is given.

pymepix.processing.rawfilesampler module

class pymepix.processing.rawfilesampler.RawFileSampler(file_name, output_file, number_of_processes=None, timewalk_file=None, cent_timewalk_file=None, progress_callback=None, clustering_args={}, dbscan_clustering=True, **kwargs)

Bases: object

bytes_from_file(chunksize=8192)

handle_lsb_time(pixdata)

handle_msb_time(pixdata)

handle_other(pixdata)

trash data which arrives before 1st timestamp data (heartbeat)

init_new_process(file)

create connections and initialize variables in new process

post_run()

pre_run()

init stuff which should only be available in new process

push_data(post=False)

run()

method which is executed in new process via multiprocessing.Process.start

saveToHDF5(output_file, raw, clusters, timeStamps, trigger1, trigger2)

pymepix.processing.rawtodisk module

class pymepix.processing.rawtodisk.Raw2Disk(context=None)

Bases: pymepix.core.log.ProcessLogger

Class for asynchronously writing raw files Intended to allow writing of raw data while minimizing impact on UDP reception reliability.

close(socket)

Close the file currently in progress. call in main below

open_file(socket, filename)

Creates a file with a given filename and path.

this doesn’t work anylonger using 2 sockets for the communication functionality needs to be put outside where you have access to the socket

write(data)

Writes data to the file. Parameter is buffer type (e.g. bytearray or memoryview)

Not sure how useful this function actually is… It completes the interface for this class but from a performance point of view it doesn’t improve things. How could this be benchmarked?

pymepix.processing.rawtodisk.main()

pymepix.processing.rawtodisk.main_process()

seperate process not strictly necessary, just to double check if this also works with multiprocessing doesn’t work for debugging

pymepix.processing.udpsampler module

class pymepix.processing.udpsampler.UdpSampler(address, longtime, chunk_size=10000, flush_timeout=0.3, input_queue=None, create_output=True, num_outputs=1, shared_output=None)

Bases: multiprocessing.context.Process, pymepix.core.log.ProcessLogger

Recieves udp packets from SPDIR

This class, creates a UDP socket connection to SPIDR and recivies the UDP packets from Timepix It them pre-processes them and sends them off for more processing

close_file

create_socket_connection(address)

Establishes a UDP connection to spidr

enable

Enables processing

Determines whether the class will perform processing, this has the result of signalling the process to terminate. If there are objects ahead of it then they will stop receiving data if an input queue is required then it will get from the queue before checking processing This is done to prevent the queue from growing when a process behind it is still working

Parameters:value (bool) – Enable value Returns:Whether the process is enabled or not Return type:bool

get_useful_packets(packet)

init_new_process()

create connections and initialize variables in new process

outfile_name

post_run()

method get’s called either at the very end of the process live or if there’s a socket timeout and raw2disk file should be closed

pre_run()

init stuff which should only be available in new process

record

Enables saving data to disk

Determines whether the class will perform processing, this has the result of signalling the process to terminate. If there are objects ahead of it then they will stop recieving data if an input queue is required then it will get from the queue before checking processing This is done to prevent the queue from growing when a process behind it is still working

Parameters:value (bool) – Enable value Returns:Whether the process should record and write to disk or not Return type:bool

run()

method which is executed in new process via multiprocessing.Process.start

pymepix.processing.udpsampler.main()

pymepix.processing.usbtrainid module

class pymepix.processing.usbtrainid.USBTrainID(name='USBTrainId')

Bases: multiprocessing.context.Process, pymepix.core.log.ProcessLogger

Class for asynchronously writing raw files Intended to allow writing of raw data while minimizing impact on UDP reception reliability

connect_device(device)

Establish connection to USB device

run()

Method to be run in sub-process; can be overridden in sub-class

pymepix.processing.usbtrainid.main()

Module contents