7 Import Data

Import grand image4 650px.png

A new generalized data import mechanism we call Data Connectors was added in Origin 2019b. As the name implies, Data Connectors create a link between your project and an external source file. The source file may be local, it can be on a company network; it can even be on a web-server. Data Connectors support options for excluding imported data when saving the project (smaller files, plus, maintain a live link to source data) and for protection of data against accidental alteration or deletion. Data Connectors also make possible import of previously-unavailable data sources, including HTML Tables and JavaScript Object Notation (JSON). Data Connectors support drag-and-drop and Auto Import on Project Open. In addition, OriginLab continues to expand Connector support to new data formats. In Origin 2021b, for instance, we have improved Connectors for National Instruments TDMS and NetCDF files; plus we've added a new Thermo Fisher Scientific MSRawFile Connector.


Other supported import mechanisms:

Contents

A Note to Upgrade Users

Longtime Origin users who are upgrading from older versions should note that import routines have moved from the File menu to a new Data menu. Here, you will also find Origin's newest data-import tool: Data Connectors.

Import fF Moved.png

Data Connectors

Data Connector General Concept.png

Data Connectors are a general mechanism for importing data to your Origin projects from local or from web-based files and pages. Data Connectors differ from older Origin import methods primarily in two default (but modifiable) behaviors:


Data Connectors support a number of common data formats, including:

To connect to SQL database, see Database.


To import data with a Data Connector:

  1. With your workbook active, click the Data menu.

Things to Know about Data Connectors

Data Connector Shortcut Menu.png
UG data connector add new.png

Origin caches data imported by Data: Connect to Web, for a period of time determined by LabTalk system variable @CFDT, plus file size. By default, a 10MB file will use cached data for ~24 hrs. In other words, if you re-import (i.e. reconnect to the web file) within 24 hrs, you will be using data that was cached on first import. If you re-import to this 10MB file after 24 hrs passes, the cache is cleared and the data are freshly imported. To learn how to control this caching behavior see, these topics:

Data Source File Path

A Data Connector icon appears in one of five states:


Regardless of the state of the connection, you can click the Connector icon, then use the Data Source File Path and Common Data Path to help you maintain the connection between your Origin files and your source data.

UG Common Data Path Dialogs.png

Data Navigator

The Data Navigator is a collapsible panel that is automatically added to the workbook when you connect to multi-page/multi-variable files such as Excel, MATLAB or Origin Projects.

UG data navigator overview.png

Data Connectors and Drag-and-Drop

There is drag-and-drop support for the file types listed under Data: Connect to File:

Connecting to Text/CSV

You can use a Data Connector to import local or web-stored text files (TXT, DAT, CSV). Data Connector methods work well for these file types, simplifying and in some cases eliminating, some of the problems associated with Origin's older X-Function-based import routines. For instance, import of date-time data always required manual configuration of date-time columns. With Connectors, if your date-time data are of a standard, easily-recognized format, that configuration is done for you automatically.


To connect to a file on a local or LAN drive:

  1. With the workbook active, choose Data: Connect to File: Text/CSV.
  2. Browse to and select your file.
  3. Set your CSV Import Options. In many cases, you can simply accept defaults and your file will be cleanly imported, so as a test, try importing data without modifying defaults. If the results are unsatisfactory, rerun the import and modify the default settings. Click the Other Options button (to the left of OK) for more options.

In this mini-tutorial, we use the Text/CSV Connector to import a text file with a date-time component. While older Origin import methods treated the date-time data as text, requiring the user to manually set the date-time column to Date after import, the Text/CSV Connector is able to automatically configure the worksheet column on import.

  1. Click the New Workbook button Button New Workbook.png and choose Data: Connect to File: Text/CSV.
  2. Browse to the <Origin Program>\Samples\Import and Export\ folder and open Date with higher precision.dat.
  3. In the CSV Import Options dialog, accept default settings and click OK. Data are imported to the workbook.
  4. Click on the column A header to select the column, then right-click and choose Properties. Note that Format has been automatically set to Date, Display has been set to Custom Display and a custom format shows in Custom Display.
Import custom date time.png

In this mini-tutorial, we use a Data Connector to import multiple files. These identically-structured files have multiple rows of metadata that we want to preserve. So, you will make use of an import filter file that has been developed using the Import Wizard, to import the files and write file metadata out to multiple column label rows in each worksheet. In so doing, you will see (a) how to import multiple files using a Data Connector and (b) how you can use an import filter file (*.oif) developed with an earlier version of Origin, to import files using Origin's newer Data Connector technology.

  1. Start a new project and with the workbook active, choose Data: Connect Multiple Files.
  2. In the files2dc dialog box, set the following:
    • Data Connector = Import Filter
    • Source = Specified Files
    • File Names = <Origin Program>\Samples\Import and Export\S15-125-03.dat, S21-235-07.dat and S32-014-04.dat
    • Same Book = enabled (checking the box will bring all files into one book)
  3. Click OK and in the dialog that opens, set Import Filter = Data Folder: VarsFromFileNameAndHeader.oif
  4. Click OK and the three files are imported as separate sheets in the workbook.


To connect to a file on the Web:

  1. Choose Data: Connect to Web.
  2. If you have a file URL, enter it in the URL box. Otherwise, set File Type to Text/CSV (optional but helps to filter Recent URLs), then click directly on Recent URLs and pick from the list.
Connect to web dialog URL.png

You can use the CSV Import Options dialog to specify a partial import of web-stored files just as you can for locally-stored files. Check the Partial Import box for guidance on the syntax used to define Column and Row import.

Note: To learn more about working with the Text/CSV Connector, see the Origin Help:

Help: Origin: Origin Help > Text/CSV Connector

Connecting to MATLAB Files

While Origin continues to support older X-Function based MAT file import, Data Connectors offer some distinct advantages for import of both locally-stored and web-stored MATLAB files:

UG slider or thumb MAT.png


To connect to a MATLAB file:

  1. With a workbook or matrixbook active, click Data: Connect to File: MATLAB or Data: Connect to Web and enter a URL to your .mat file.
  2. Select the variables that you want to import (add them to the Browser's bottom panel) and click OK.


Note: To learn more about working with the JSON/MATLAB Connector, see the Origin Help:

Help: Origin: Origin Help > JSON/MATLAB

Connecting to NetCDF and Shapefiles

The NetCDF file format is used for sharing multi-dimensional scientific data. The format is widely used in climate science. Shapefiles are commonly used to store geographic boundaries and other geo-spatial data.

Import NetCDF Image Map.png


To connect to a NetCDF file:

  1. With a workbook or matrixbook active, click Data: Connect to File: NetCDF; or Data: Connect to Web and enter a URL to your .nc file.
  2. Select the variables that you want to import by adding them to the Browser's bottom panel.
  3. Adjust Import Options ("pencil" button) as needed and and click OK twice.
  4. If a workbook is active but the variable requires a matrixbook, you will be prompted to add a new matrixbook or an embedded matrixbook in the active workook.


To connect to a Shapefile:

  1. With a workbook or matrixbook active, click Data: Connect to File: Shapefile and browse to and choose your .shp file.
  2. Specify whether to Load Attributes and (optionally) encoding and/or Convert to WGS84, and click OK.
Note: To learn more about working with the NetCDF Connector, see the Origin Help:

Help: Origin: Origin Help > NetCDF Connector

Connecting to a Cloud Drive

You can connect to Origin, MS Excel and ASCII files stored on MS OneDrive or Google Drive. Once you've established your account credentials, connecting to cloud files is similar to connecting to local files:

  1. With a workbook active, click Data: Connect to Cloud. This opens the browser for picking files.
  2. Use the browser's drop-down controls to switch cloud services or accounts, if needed.
    Data Connect to Cloud Browser.png
  3. Pick a file and click Open.

Note that the first time you connect to a cloud account, you'll be asked to verify that account. OriginLab will only access and transfer data between your cloud account and your local computer using a temporary file. No other communication of information occurs.

Working with Excel Data

There are several ways to work with MS Excel data in Origin:


Things to know:


Note: To learn more about working with Excel data, see the Origin Help:

Help: Origin: Origin Help > Working with Microsoft Excel and Connecting to File > Excel

Both the Excel Connector and impMSExcel routines support partial import of Excel data. For more information, see the Origin Help file topic Partial Importing and Re-importing.

Connecting to Excel Files

Origin's Data Connectors create a live link between your external data files and your Origin project.

Excel connect import options structure.png
  1. Click the New Workbook Button New Workbook.png button to create a new workbook, then choose Data: Connect to File: Excel.
  2. Browse to and open <Origin Program>\Samples\Import and Export\United States Energy (1980-2013).xls.
  3. In the Excel Import Options dialog that opens, accept default settings and click OK.
  4. In the Data Navigator panel that opens to the left side of the workbook, hold the CTRL key and select the other three (dimmed) sheets, right-click and choose Add and Connect Sheet(s). The remaining sheets -- Natural Gas, Coal and Total Primary Energy -- are imported into the workbook. Note that if the unconnected sheets are large in size, you can opt to connect to them one-at-a-time, as needed.

Import from File

The import routines listed under Data: Import From File pre-date Data Connectors. While the Import From File routines are no longer being developed, they are still supported.

Note, also, that Connectors may not exist for some of the file types listed in this table. If no Data Connector exists, you should use the Data: Import From File methods.

The Standard toolbar Import Single ASCII button Button Import ASCII.png now calls the Data: Connect to Files: Text/CSV routine. For the impASC routine, see the next section.

Importing ASCII Files with impASC

Import data 1.png
Note: To learn more about working with the ASCII Import using the impASC dialog, see the Origin Help:

Help: Origin: Origin Help > Customizing the ASCII Import Options

Saving Import Settings

Once you've adjusted impASC dialog import settings, you can save them in one of two ways:

Saving import settings with the worksheet template offers portability (e.g. templates are easily shared with colleagues). Saving import settings as a Theme offers the advantage of being able to recall any number of import options combinations (Themes) when importing via the impASC dialog box.

For more information on Themes and templates, see the Customizing Origin chapter later in this User Guide. Additionally, see topics listed under "Topics for Further Reading", below.

You can quickly load your custom import theme from the list at the bottom of the Data menu. You can also use the Data: Import from File: Add/Remove File Types ... menu to open the Import Menu Customization dialog box where you can select and add User Themes to the Data: Import From File menu.

Third Party Formats

If a Data Connector does not exist for a certain third-party format, you can still use Origin's older X-Function-based import routines. These include many widely-used formats, including pClamp, NI TDM, and NetCDF. As with all X-Function-based routines, you can customize third-party file import options and save your settings with the worksheet template or as a dialog Theme for later use. As with other supported file types, you can import third party files by drag-and-drop.

File Type Method To Learn More, see

Help: Origin: Origin Help > Importing and Exporting Data


Thermo (SPC, CGM)
pCLAMP (ABF, DAT, AB?)
Princeton Instruments (SPE)
Sound (WAV)
NetCDF (NC)
CDF (CDF)
HDF5 (H5, HE, HDF5)
NI DIAdem (DAT)
Famos (DAT, RAW)
ETAS INCA MDF (DAT, MDF)
IgorPro (PXP, IBW)
Prism (PZFX, XML)
MATLAB (Mat)
Minitab (MTW, MPJ)
KaleidaGraph (QDA)
SigmaPlot (JNB)
MZXML(mzData, mzXML, mzML, imzML)
EarthProbe (EPA)
EDF (EDF, BDF, REC, HYP)
Somat SIE (SIE)
JCAMP-DX (DX, DX1, JDX, JCM)
HEKA (DAT)
SPSS (SAV)
BRUKER OPUS
SAS (sas7bdat)
Tektronix (ISF)
Data Translation (DCF, HPF)
Binary 2D Array (IMG, B2D)
Image

Data: Import from File: > Importing Third Party Files
Note: Some of these 3rd party file formats are hidden by default. To add, choose Data: Import From File: Add/Remove File Types.... This opens the Import Menu Customization dialog box where you can choose which file types to show in the Import From File menu. To insert a separator after a menu command, right-click on the command in the right-hand panel and choose Add Separator. To remove a separator, double-click on it in the right-hand panel.

If no Data Connector exists for a given file type (i.e. those listed in the table above), then drag-and-drop support is managed by file extension, using the Filter Manager dialog. From the main menu, click Preferences: Import Filters Manager....

Note: To learn more about working with the Third-party Formats, see the Origin Help:

Help: Origin: Origin Help > Importing Third Party Files

The Import Wizard

For import of complex ASCII files -- those not satisfactorily handled by the methods previously discussed -- you can use the interactive Import Wizard. Files with an optional header section followed by a sequential binary structure, can also be imported using the Import Wizard:

Another advantage to using the Import Wizard to your ASCII data is that you can create import filters that are targeted to a group of similarly-named data files. This is especially helpful when those files contain metadata that that needs to be handled in a certain way. The filter can then be used for drag-and-drop import operations and it can be added to the Data: Import From File menu. You can view and edit these import filters by Preferences: Import Filters Manager menu.


To run the Import Wizard:

  1. Choose Data: Import From File: Import Wizard... or click the Button Import Wizard.png button located on the Standard toolbar.


Note: For more information, view the help topic:

Help: Origin: Origin Help > Importing and Exporting Data > The Import Wizard

Copy/Paste the Clipboard Data

One obvious way to get data into Origin is simply to copy data to the Clipboard and paste it into Origin. This can be as easy as selecting a block of data in an Excel workbook and pasting it into an Origin workbook. However, Clipboard data may not be formatted, arrayed or delimited in a way that makes it easy to paste your data into Origin. Here are some things you should know:

Import data wizard composite resample office 600px.png

Selectively Import Data by Querying an Origin File

You can selectively import columns of data from an external Origin file by specifying an SQL-like Query in the Data Connector Browser prior to import.

UG Import Origin SQL.png

To selectively query Origin files using the Origin Connector:

  1. With the worksheet active, choose Data: Connect to File or Connect to Web.
  2. Browse to your Origin file and click Open. This opens the Data Connector Browser.
  3. Click the Select Query button UG Select Query Button.png.
  4. Enter your Query in the Select Query dialog, then click OK.

Data will be selectively imported into the active workbook as specified by your query.


Common Keywords:

Keyword Description
_SName short name, e.g. _SName, Sheet_SName
_LName long name, e.g. _LName, Book_LName
_Comments comments, e.g. _Comments, Sheet_Comments
Column_D1 column user-defined parameter, where 1 is the parameter index
Column_[Name] column user-defined parameter, where [Name] is a parameter name containing spaces


Examples:

Select Column from Project where _SName == "A";
Select Column from Project where _Comments like "S*3";
Select Column from Project where (Column_P1 like "*y*" and Column_D1 > 3);
Select Column from Project where Sheet_SName like "Trial*";
Select Column from Project where (Book_SName like "Book*" or Book_LName like "*fit*")


For more information, see Examples of using SQL query condition in Origin Connector.

Database

Origin supports connecting to databases by ODBC and OLE DB. Additionally, Origin includes a simple SQL Editor to add connection strings and generate SQL queries within Origin.


To import SQL data, start with the Data: Connect to Database menu command:

UG database connector dialog.png


Other things to note:

DB Save wo data.png
Note: For more information, see the Origin Help

Help: Origin: Origin Help > Importing and Exporting Data > Importing Data from a Database

Digitizer

At times, you may only have a graphic representation of some data that you would like to explore further. Use the Digitizer to import an image file, then digitize the plots, converting your image to numeric data that you can study and re-plot using Origin's analysis and graphing tools.

Digitizer data import chapter 2.png


To open the Digitizer:

  1. Click the Digitize Image Button Digitize Image.png button on the Standard toolbar or choose Tools: Digitizer... from the main menu.

Saving Origin Files without Imported Data

Users have long had the option of saving Origin files without data. Save Template As..., Save Workbook As Analysis Template and Save Project without Data, were three options for saving Origin files without data while preserving settings and operations with the file.

Users now have the option to save Origin files without imported data. This is a particularly attractive option for those who work with large data files, where saving of that data with the file would hinder performance or waste disk space.

Exclude imported when saving.png
Exclude imported when saving Properties dialog.png


The act of excluding data when saving differs from saving a template file in a couple of important ways:

Re-Importing Data Files

There are two import commands in the Data menu that are specifically intended for repeat import of the same file (i.e. the file path and name must be the same):

Notes:

  • Only one file can be re-imported at a time. When you use either of these two menu commands, only the file associated with the active worksheet or matrixsheet, will be imported. If the selected data are imported from multiple files, re-importing will fail.
  • If you import an ASCII file via the Import Wizard and select Data: Re-Import..., the impASC dialog will open instead of the Import Wizard.
  • Re-import does not work for Binary or User Defined data types brought in using the Import Wizard.

Clone Import

Clone import works by "modeling" an import -- and optionally, an associated analysis operation -- and then applying what was done to the initial file, to a series of similar files. As long as the active book contains imported data and associated operations, it can be cloned. In this sense, cloning is similar to creation of an Analysis Template but is somewhat more limited in power (e.g cannot be used with the Batch Process tool).

Clone import is supported by both Data Connector and X-Function-based import methods. A brief tutorial will show you the basics of cloning import and analysis operations.

  1. Click the New Workbook button Button New Workbook.png to open a new workbook.
  2. Using Window's File Explorer, browse to your <Origin Program>\Samples\Curve Fitting folder and find a file called Sensor01.dat. Drag-and-drop the file onto the new workbook. The .dat file is imported using the Text/CSV Connector.
  3. Click the column B header to select the column, then click the Scatter button Button Scatter.png to create a scatter plot.
  4. Click on the X or Y graph axis scale and in the Axis dialog box that opens, click the Scale tab, then press Ctrl and select both Horizontal and Vertical axis icons in the left panel. Set Rescale to Auto and click OK.
  5. With the graph active, click Analysis: Fitting: Linear Fit and open the dialog box. Set Recalculate = Auto, accept other default settings and click OK. Answer No to the Reminder Message and click OK.
  6. Right-click on one of your workbook tabs and click Add Graph as Sheet. This will add your scatter plot with the linear fit results, as a page in your workbook.
    UG clone import add graph.png
  7. Make sure the sheet with data imported from Sensor01.dat is active and click the Clone Import button Button Clone Import.png on the Import toolbar.
  8. Browse, once again, to your <Origin Program>\Samples\Curve Fitting folder, press Ctrl or Shift and select files Sensor02.dat - Sensor07.dat. Click Add Files and then click OK. What you should see is that using the original book as a model for import and analysis operations, the Sensor files are imported using the Text/CSV Connector and, one-by-one, they are processed (it may take a few seconds for all operations to go to completion).
    UG clone import results.png

Topics for Further Reading