Thursday, January 28, 2010

New blog posts on ddj.com

I am writing today to mention that two new blog posts are available on www.ddj.com/go-parallel relating to using a QuickThread two stage parallel pipeline to improve the performance of the input end and output end of an application (as opposed to 3 stage design.

The typical 3-stage pipeline is:

MyPipelineIOContext pio;
pio.Init(your init args here);
qtPipeline< MyPipelineIOContext, MyPipelineBuffer > pipeline;
pipeline.addPipe(InputPipe);
pipeline.addPipe(DoWorkPipe);
pipeline.addPipe(OutputPipe);
pipeline.run(&pio);
pipeline.WaitTillDone();

The 3-stage pipeline works quite well for most applications. However, there are certain classes of applications where a single linear pipeline is unsuitable. One such example was referenced in the ddj.com/go-parallel blog. For this you might find sandwidching your work phase between two 2-stage pipelines:

// 2-stage pipeline to suck-in and prep data
MyInputPipelineIOContext pio_in;
pio_in.Init(your init args here);
qtPipeline< MyInputPipelineIOContext, MyInputPipelineBuffer > pipeline_in;
pipeline_in.addPipe(InputPipe); // I/O
pipeline_in.addPipe(DoInputPrepWorkPipe); // compute
pipeline_in.run(&pio_in);
pipeline_in.WaitTillDone();
// input complete (back to serial code)

// start work (which has parallel constructs)
DoWorkOutsideOfPipeline();
// done with work

// use 2-stage pipeline to output results
MyOutputPipelineIOContext pio_out;
pio_out.Init(your init args here);
qtPipeline< MyOutputPipelineIOContext, MyOutputPipelineBuffer > pipeline_out;
pipeline_out.addPipe(DoOutputPrepWorkPipe); // compute
pipeline_out.addPipe(OutputPipe); // I/O
pipeline_out.run(&pio_out);
pipeline_out.WaitTillDone();

The prep work for input might consist of converting ASCII text input into binary format. The prep work for output migh be converting binary format to ASCII text. Although you are free to use as you see fit.

Jim