The data source lets you choose which columns or tables of data to import. However, depending on the spreadsheet or database that is providing the information, a typical import can bring in thousands of records and tens of columns. You need to examine the data that’s been imported, make decisions about the data type, evaluate what to do with it, and do general data cleaning.
When Adaptive Integration imports the information from the data source, it puts the extracted data in a staging area. The staging area is a large buffer in Adaptive Integration that holds the imported data. You can use the staging area to preview and browse the data you’ve extracted, combine or add columns, and apply business rules to filter and manipulate imported data. You can also choose to configure the data import to extract data as an incremental or a full import. When you are working with tables, you can look at the data in a spreadsheet or database directly, or you can view the data in the staging area.
Data in the staging area tends to stay until you delete it or update it. This means that you don’t need to go to the data source to re-import data every time; you can use the archived data in the staging area if it’s recent enough. You should avoid doing a full refresh on the staging area for a data source frequently because of the quantity of data you’re likely to be extracting, but you should update the information with incremental (fresh) data if there have been changes to the information in the spreadsheet or database.
If you have multiple data sources, the information from each one stays separate. You can think of the staging area as a big “scratch” area, where the raw data is imported and can be previewed, prepared, and evaluated.