Quite often you may want to process data from multiple files all at the same time. There are a number of options available to you if you want to do this.
1. Have a seperate source adapter for each file and amalgamate them together with the Union All component.
Advantages of this approach:
- This works well if the files are of differing formats
- You only have one executable in order to do everything
Disadvantages:
- This does not work if the number of files differs each time
2. Loop over the files in a Foreach loop and append to a raw file. Then, process all data together in a seperate data flow.
Advantages:
- You can re-use data-flows for files of the same format
- Works regardless of the number of files
Disadvantages
- You have more executables in your package
- You will need to build control-flow logic to handle differing formats or else have some clever logic in your data-flow inside the Foreach loop
- You need to stage the data (in the raw file)
3. Use a MULTIFLATFILE connection manager
Advantages
- Only one executable in your package
- One set of logic if all files are of the same format
- Works regardless of the number of files
Disadvantages
- You need some clever logic in your data-flow to handle differing file formats
Its clear that you choice depends on your circumstances so given that I don't know your circumstances I won't make a recommendation.
-Jamie