A pipeline lookup transformation uses a source qualifier as its source. It sources data from a source qualifier in a separate pipeline in the mapping.
In this type of lookup you create an additional pipeline from the lookup source using a source qualifier. This pipeline does not contain any other transformation and is also called a partial pipeline.
After this you just need to set the source type attribute to be the source qualifier in the session properties of the lookup transformation. The lookup cache will get populated from the result of the source qualifier.
When to use this type of lookup?
1. Use it when the lookup source is not a relational or flat file.
2. When you need to improve the performance of the mapping having lookups on relational or flat file sources as you can utilise partitioning to speed up the lookup cache creation via source qualifier pipeline.
Things to remember when using this type of lookup arrangement
1. You cannot configure the target load plan of the partial pipeline
2. Do not enable High Availablity recovery for sessions that have real time sources for pipeline lookup as it may produce unpredictable results.
Leave a comment if you need any more details regarding the configuration of such mappings
Hi Raghav,
Thanks for the excellent explaination on this , But i would like to know how to enable mulitple partitions for better performance .
Thanks in advance !
Thanks again for this .
Hi Raghav,
Can you be a bit more specific in describing the pipeline concept?
I’m really getting confused between lookup source and source qualifier.