Data Flow Steps

This topic lists the steps that you can add to a BI Server data flow, grouped as they appear in the step menu, with the properties of each step and their default values. For more information about data flows, see Understanding Data Flows. For a worked procedure on a data source step, see Configuring a Historical Alarms Step.

Common Properties

These panels appear on more than one step.

Panel or property	Description
Data Sources	The list of points that a data source step reads, with one Point Name per row. Click here to add new item adds a row, and Click to add multiple tags opens the data browser to add several points at once. Select Use Parameter to take the point name from a data flow parameter instead. On the steps that allow it, a point name can contain the wildcards `*` and `?`, or a regular expression that starts with `^`, to read several points at once. Steps that read assets show this panel as Asset Path.
Time Settings	The Start Time and End Time of the period that the step reads. The default start time is the current date and time, and the default end time is one hour later. The start time must be earlier than the end time. Each has a Use Parameter check box.
Value Data Type	The data type of the Value column that a step produces. The default, Native, sets the type of the column from the first value received while each value keeps its own type. Selecting a data type converts every value to it; a value that cannot be converted gets a bad quality.
Use Parameter	Binds the property next to it to a parameter defined on the Parameters tab of the data flow. Learn more

Data Sources

The step names that include a product name follow the name of the product on your system.

Step	Properties	Description
Data Historian Raw Data	Data Sources (wildcards allowed) Time Settings Value Data Type (default: Native)	Reads the raw historical values of the points in the time range, one row per value.
Data Historian Aggregated Data	Data Sources (wildcards allowed) Time Settings Aggregate Name—the aggregate to calculate, such as Average, Minimum, Maximum, Total, Count, Start, End, Delta, Percent Good, or Percent Bad Processing Interval—the length of each aggregation interval Time Zone (default: UTC)—when the processing interval is longer than one hour, the adjustment rules of this time zone compensate aggregates calculated over daylight saving transition days Percent Data Good (default: 100)—the minimum percentage of good data in an interval for the interval's status to be Good Percent Data Bad (default: 0)—the maximum percentage of bad data in an interval above which the interval's status is Bad Treat uncertain as bad (default: selected)—whether values with an Uncertain status count as bad or as good in the calculation Value Data Type (default: Native)	Reads aggregated historical values of the points, one row per point and interval.
Historical Alarms	Data Sources Time Settings Event Fields (default: empty)—the event fields to read, as `EventType.Field` entries separated by semicolons; the Event Fields Editor button opens the field picker. When the list is empty, the step asks the source for its fields at run time and reads all of them. Qualify Column Names (default: selected)—name the output columns `EventType.Field`; clear it to use the field name alone	Reads the alarm and event history of the sources in the time range, one row per event and one column per event field. Learn more
Fault Detection	Asset Path (wildcards allowed) FDD Data Source—Causes, Incidents, or Latest Causes Fault Name Time Settings	Reads fault detection data of the selected kind for the faults of the assets under the asset path.
Grid Point Builder	Point Name—the Database Connector data source A table of the data source's parameters with Parameter Name and Default Value, each with Use Parameter	Builds the point name of a parameterized Database Connector data source from the parameter values and reads the resulting dataset.
Dataset	Data Sources	Reads any GENESIS point that returns a dataset.
Real-time	Data Sources Read batch size (default: 500)—the maximum number of points read concurrently; further points wait in a queue until a read in the batch completes, and the order of the reads is not guaranteed Read timeout (default: 60 seconds)—the timeout of each read; a point whose read times out has a null value and a bad quality in the output A bad quality delay (default: 0)—when greater than zero, how long to wait for a good quality update after a bad quality value before sending the bad value to the output Batch read delay (default: 0)—when greater than zero, how long to wait before processing the next batch Value Data Type (default: Native)	Reads the current value of each point once, one row per point.
Asset Property Values	Asset Path (wildcards allowed) Property Name Filter (default: (null), no filter)—wildcards `*` and `?`, or a regular expression that starts with `^`, to match property names Property-Read batch size (default: 500) Property-Read timeout (default: 5 seconds) A bad quality delay (default: 0) Batch read delay (default: 0)	Reads the values of the properties of the assets under the asset path. The batch size, timeout, and delays work as in the Real-time step.

Dimensions

Step	Properties	Description
Assets	Asset Path (wildcards allowed) Use compatibility schema (default: cleared)—produce the same output columns as previous versions of the step	Lists the assets under the asset path, with an AssetPath column and one column per level of the asset hierarchy. Learn more
Asset Properties	Asset Path (wildcards allowed) Property Name Filter (default: (null), no filter)	Lists the names of the properties of the assets under the asset path.
Historical Tags	Data Sources (folders and points, wildcards allowed)	Lists the Data Historian tags that match the configured folders and points, with a PointName column. Learn more
Time	Time Settings Resolution (default: 1 hour)	Generates a date and time dimension: the intervals between the start time and the end time at the configured resolution.

Shaping Steps

These steps work on the columns produced by the steps before them.

Step	Properties	Description
Parse JSON	JSON column name Column names delimiter character (default: `.`) Use original property names as prefix (default: cleared) Map parsed properties to existing columns when possible (default: cleared)	Parses the JavaScript Object Notation (JSON) text in a column into one column per property.
Parse GPS Location	GPS Location column name Include Altitude (default: cleared) Use original property names as prefix (default: cleared) Column names delimiter character (default: `.`)	Parses a Global Positioning System (GPS) location column into its coordinate columns, optionally including the altitude.
Add Column	Column Name Expression	Adds a calculated column. The expression can use the existing columns as variables.
Rename Column	Select Column to rename New Column Name	Renames a column.
Transform Column	Column Name Expression	Replaces the values of a column with the result of the expression, for example to round numeric values with the roundto function. Learn more
Change Column Type	A table of the columns with Column Name, Current Data Type, and Select New Data Type	Converts the values of the selected columns to the new data type.
Remove Column	Available Columns—clear a column to remove it Invert selection	Removes the cleared columns from the output. The columns have already been read from the source; to avoid reading them at all, narrow the request in the data source step where it allows it.
Split Column	Column To Split Column Split Type—Delimiter, Character to digits, Digits to characters, or Number of characters Delimiter (for the Delimiter type)—Comma, Semicolon, Colon, Space, Tab, Equals sign, or Custom Number of characters (default: 1, for the Number of characters type) Split options (default: Repeat)—Repeat keeps all the results of the split, Leftmost keeps only the leftmost result, Rightmost keeps only the rightmost result Sample Size (default: 1000)—how many rows to examine to determine into how many columns the source column is split	Splits a text column into several columns. With Delimiter, the column is split on every occurrence of the delimiter. With Character to digits, it is split where a character is followed by a digit, so `Floor01` becomes `Floor` and `01`. With Digits to characters, it is split where a digit is followed by a character, so `01Floor` becomes `01` and `Floor`. With Number of characters, it is split into columns of that many characters each, and the last column holds the remainder.
PIN Column	Columns to pin to the output Invert selection	Pins the selected columns so that they are propagated to the output of the data flow.
Filter	Expression	Keeps only the rows for which the expression is true.
Transpose	Pivot Column Name Values Column Name Sorting Column Name (optional) Sort by descending (default: cleared)	Produces one column for each distinct value of the pivot column, filled with the values of the values column. A large number of distinct values slows the preview down.

The Transpose step must consume its entire input before it produces any output. When a data table loads from a data flow that uses it, the DataLoadTimeoutSec parameter of the BI Server Point Manager must be long enough for the step to read the whole input. Learn more