In Dataverse, the key buildings blocks for easily building your logic or your "data flow" are the nodes and functions that come pre-built into the product.
- A node is an individual piece of functionality that applies logic to the underlying data at that particular step in the data flow.
- A function is a pre-built instruction that performs an operation and returns a value.
But with so many nodes, functions and more, where do you start?
This article directs you towards some commonly used nodes, functions and operators to help you get started.
Commonly Used Nodes
By default, a small subset of the available nodes is displayed, which is ideal for new users who are just getting started creating data flows. This subset is called Favorites, and consists of a few of the most commonly used nodes. Substantial data flows can be created from just this subset.
Hover over the node to see a brief description of what it does:
Choose All Nodes from the drop-down list to see all of the nodes which are available for creating your data flow.
Code-free Data Preparation
A range of common data preparation tasks can be performed in the Data Viewer without having to use any coding. You can sort and filter data using the context menu for a data field.
The filter options that are displayed depend on the field's data type.
When a Filter or Sort has been applied to the data, you can add it to the canvas as a pre-configured Filter/ Split or Sort node.
If required you can also manually add a Filter, Split or Sort node to the canvas and then use the grid editor to configure the node without having to write any code.
You can customize the operation of a Filter or Split node clicking the Advanced button, which allows you to view and edit the underlying Python code used by the node.
Even if you are not yet at the stage where you want to write your own custom filter expressions viewing the code can be a helpful learning exercise to familiarize you with using Python in Dataverse.
Using the Python Language
Python has superseded the proprietary Dataverse Scripting that was used by earlier versions of Dataverse. Dataverse Script is still available within the product but Lavastorm recommends the use of the Python language for current projects.
There are many resources available online that can help familiarize you with the Python such as
The Dataverse Help provides information on Dataverse-specific functions and how we bind records and metadata, and details of the custom Python modules that are provided with Dataverse. See the Python Scripting section of the embedded Help or the corresponding section in the online documentation here.
Commonly Used Proprietary Dataverse Script Functions
As previously mentioned, the proprietary Dataverse Script language has been superseded but is still retained for users who are transitioning to Dataverse from our earlier Lavastorm product or users who may need to modify existing data flows to use the newer Python-based nodes in Dataverse. Below are some commonly used functions and operators. You can also access further information in the Dataverse Script section of the Dataverse Help or in the online documents here.
Converting data types
Note: Many of the data conversions can be easily performed using the Data Analyzer or Data Converter nodes.
|double||Can be used to convert a field to double data type. Example: "1.03".double() --> 1.03|
|int||Can be used to convert a field to integer data type. Example: "3".int() --> 3|
|long||Can be used to convert a field to long data type. Example: "100600".long() --> 100600|
|str||Can be used to convert a field to string data type. Example: str(400) --> "400"|
|date||Can be used to convert a field to date data type or obtain current date.
Examples:"2016-01-27".date("CCYY-MM-DD") and "01/27/2017".date("DD/MM/CCYY")
Note that the input to the date function needs to represent the format that you receive the data. Please refer to the Help page for more information.
|time||Can be used to convert a field to time data type or obtain current time. Example: "09:14:00".time("HH:MM:SS")|
Working with String values
|strlen||Calculates the length of your input string.|
|toLower||Convert characters in a string to lower case|
|toUpper||Convert characters in a string to upper case|
|trim||Removes leading and trailing white space|
|left||Gets the first N characters of the input string|
|right||Gets the last N characters of the input string|
|substr||Gets a substring of the input string|
|replace||Use to perform a find-and-replace in your input string|
Aggregating data (use within Agg Ex node)
|groupSum||Totals the value amounts of a specified field for the selected group, for example, summarize the total Amount by Product Type.|
|groupCount||Counts the number of instances of a value in a specified field for the selected group, for example, the count of Sales Orders by Date.|
Checking for Null values
Note: The Data Analyzer node is a great way to identify whether any column fields have NULL values.
|isNull||Evaluates to true if a value is NULL and false otherwise.|
|isNotNull||Opposite of above: Evaluates to true if a value is not NULL and false otherwise.|
Commonly used Operators
|A conditional statement. If the predicate evaluates to true, then execute the consequent statement or statements. Otherwise, execute the alternate statement or statements|
|where||Specify a condition to control when to output certain data|
You can find node examples installed with your Dataverse instance:
To view the examples in Dataverse, please import the *.lna file by following the instructions found in Dataverse Help > Advanced topics > Sharing data flows and library nodes