The SDSAI Iterator Expression Language

This software is available under the MIT license. See LICENSE for more information.

ItrEx - A simple evaluator

ItrEx specifies and implements an evaluator. That is, it doesn’t particularly care what the language is that captures the data from the human into text, it only cares about taking a Java List object and evaluating it to a value.

ItrEx doesn’t try to be a general purpose language. This is important. The developer that chooses to use ItrEx will develop it into a language to suit their use case, adding functions that have meaning to them. ItrEx does provide a few basic functions, but mostly to support evaluation.

IterEx API

General Calls

[version]

Returns the version of the API.

[import <string or class>]

Import functions into the root evaluation context. If the argument is a string, the class it points to is loaded. If the class is a Package class, then its importTo function is called. If the class is another type, then all static members that are FunctionInterface types are registered as functions.

If you include as <package_name> the package will be imported under the given package_name. So [import my.long.package.name as mypackage] puts the functions such as foo under mypackage.foo.

[evalItrml <file>]

Load the given file into memory, parse it as an .itrml file and evaluate it in the current evaluation context. If you use this to load custom functions, realize that this must be evaluated under the root context or the function registrations will be discarded.

[dict.mk ke1 val1 key2 val2]

Make a dictionary.

[dict.put dict key1 val1]

Put a value to a dictionary. Returns the dictonary.

[dict.get dict key1 default]

Get a value from a dictionary. If the key is undefined the default is returned or null if no default is given.

Function

[arg]

Pull one argument value off the argument list passed to the evaluation context. This is how a function should get its arguments. This will cause a NoSuchElementException to be thrown. Make sure [hasArg] is true before calling this.

[args]

Return all arguments passed to the evalution context as an iterator.

[fn <name> <optional body>]

If no body argument is provided [fn] will look up a defined function and return it. If no function is defined for the given name and exception is thrown.

If a body is given [fn] will define the function and register it. The funtion will still be returned. This usage is equivalent to [register foo [function [...]]].

[function [...]

Construct a function that can be called or used. See the [args], [arg] and [hasArg] functions.

[let
    [set f [function [if [hasArg] [arg] []]]]
    [[get f] hi]
]
[hasArg]

Returns true if the evaluation context has an argument remaining. Function calls receive a new argument list.

[hashArgs]

All arguments that are strings of the pattern name:value will be converted into environment values in the following way. The string is split on the ":" into two strings. The first string is used as the variable name and the second string is used as the value.

So the argument "a:hello" will cause [get a] to return the string "hello".

Any null argument is discarded.

Any argument that is not mapped to an environment value is retained as an argument. The new iterator of argument values is also returned by [hashArgs].

[nameArgs name1 name2...nameN]

Name each argument to the function with the given names. Naming is done by fetching the value from the argument iterator list (making it unavailable to calls to [arg]) and stores it in the evaluation context under the name.

For example the code

[[function [nameArgs a b c]] 1 2 3 4]

will cause [get a] to be 1, [get b] to be 2, [get c] to be 3 and 4 is left on the array list such that a call to [arg] will return 4.

The case of

[[function [nameArgs a b]] 1]

will set a = 1 but leave b unset.

[register <name> <function>]

Register the given function under the given name.

A typical usage would be

[register f [function [toString [arg]]]]

This registers the funciton under the name f. The reason to separate register from function is to allow for constructions such as [register f [curry...]] and [register f [compose...]].

Functional

[callFlattened foo [args...]

Call the function, foo with the given arguments, flattened. That is, any argument in the args list that is a list has its individual elements returned as arguments to foo. Any argument in the args list that is an iterable will have its individual elements returned as arguments to foo. Other arguments are directly passed as elements to foo.

This allows for

[callFlattened foo [list a b] [list [list c d]]]

to be constructed which will result in an execution equivalent to

[foo a b [list c d]]

This is used, for instance, when [map]'ing lists into arguments that must be mixed with other arguments to function calls.

[compose <function 1> <function 2>...]

Compose 1 or more functions (not function names) into a into a single function. Each function should take only 1 argument, that returned from the following function. The last function may take any number of arguments. The compose function does not take function names, but functions. Use curry with no arguments to fetch functions by name.

Curry Example: This runs f(g("hello")).
[[compose [curry f] [curry g]] "hello"]
[curry <function name> [args...]]

Return a new function that will call the function name with the given arguments and any additional arguments passed to the returned function.

[foldLeft <function 1> <initial> [elements...]]

This is aliased as fold. Fold the elements in the list of elements using function 1 with the given initial value. The function 1 function should take two arguments. The first is the folded value (starting with the initial value) and the second argument is an element in the given list of elements.

[pipeline <function or function name>]

Given functions or function names, this will return a function that passes the result of the first function to the second function as an argument, and so on. The effect is that the functions for a processing pipeline. This is similar to [compose] but instead of f(g(x)) this results in g(f(x)). Note that [pipeline] executes procedurally in order while [compose] will execute its functions recursively in reverse order. This typically only matters for scoping variables.

Pipeline Example: This runs g(f("hello")).
[[pipeline [curry f] [curry g]] "hello"]

List

[map <function> <iterator>]

Return an iterator that maps the elements from iterator to the result of applying function to those elements. The elements from the argument iterator are not mapped using function until they are requested from the returned iterator.

[mapFlat <function> <iterator>]

The function [map] passed the arguments to the function. This function flattens the arguments before passing them to the function. This is equivalent to

    [map [curry callFlattened [curry someFunction]] [arg]]
[head <iterator>]

Return the first element.

[tail <iterator>]

Consume the first element and return the remaining iterator.

[last <iterator>]

Evaluate every argument and return the result of the last one.

[list a b c…​]

Evaluate all arguments and put the results into a list.

[listFlatten <iterator 1> <iterator 2>…​]

Take a list of iterators and flatten all elements into a list. If a non-list item is encountered it is directly added to the list. This is more tolerant than the flatten function.

[filter <function> <iterator>]

Filter the input iterator using the given function as a predicate. Filtering is done by pre-fetching elements from the input iterator until the predicate returns true for that element. When another element is called for, the current element is returned and the next one is fetched.

[flatten <iterator 1> <iterator 2>…​]

Takes a list of iterators. Returns an iterator that will walk through elements of each of those argument iterators. Unlike listFlatten, this does not materialize the inputs into a list, allowing for memory savings.

[flatten2 < <iterator1>, <iterator2> >, < <iterator3>, <iterator4> >

Just flatten will take iterators and concatenate them. However, when dealing with the output of something like a call to [map] you can easily end up with a single iterator that contains iterators. In this case, flatten would just return that single iterator with no change. What we really want is a way to unwrap the outer iterator and concatenate the inner elements. Flatten2 does this. It is equivalent to a call to [callFlatten [curry flatten]] …​].

Flatten2 Example: This returns the iterator [1, 2, 3, 4, 5, 6]
[flatten2 [list [list 1 2] [list 3, 4]] [list 5 6]]

String

[string.join joinString string1 string2…​]

Takes 1 or more strings. Returns a string joined by the first string. If this encounters an iterator as an argument it will drain the iterator, joining each of those elements as a string.

[string.split splitPattern string]

Split the second string using the first string as a regular expression.

[string.concat string1 string2]

Concatenate all arguments as strings.

Casting

[string arg]

Return the result of calling toString() on the argument.

[int arg]

Convert the argument to an integer.

[float arg]

Convert the argument to a float.

[long arg]

Convert the argument to a long.

[double arg]

Convert the argument to a double.

[boolean arg]

Convert the argument to a boolean.

Printing

[help <function>]

Print help text for a function, if any.

[print …​]

Collect all its arguments into a list and print them as they are collected. That list is then returned as an iterator.

The difference between trace and print functions is that print marshals all arguments into a list and prints them, and so will pay the memory cost to store those arguments. Trace only prints arguments as they pass by when called for by the parent function.

[printErr …​]

Like print but uses standard error.

[trace <function> args…​]

Print the function and each of the arguments. After the arguments are each evaluated and printed, they are then passed to the function.

This should allow any function call to be prefixed with trace and result in helpful output. The one drawback is that the lazy evaluation of input arguments is lost. For modest lists of arguments this is not an issue.

[traceErr …​]

Like trace but uses standard error.

Logging

[log.debug …​]

Log all arguments at DEBUG. This is very similar to print.

[log.info …​]

Log all arguments at INFO. This is very similar to print.

[log.warn …​]

Log all arguments at WARN. This is very similar to print.

[log.error …​]

Log all arguments at ERROR. This is very similar to print.

Looping / Iteration

[for <name> <iterable> <body>]

For sets name to each value in iterable. It will then evaluate the body over and over, for each value in iterable. The last evaluated value of body is returned. Because for needs to store the body unevaluated it must not be directly curried or composed as that proxies the argument list inside the evaluation engine.

Returns the sum of 1, 2, 3 and 4.
[last
   [set i 0]
   [for j [list 1 2 3 4] [set i [add [get i] [get j]]]]]
[range [start] <stop> [step]]

Return an iterator that will walk from the start to the stop by adding the step value. If 1 arguments i passed, it is treated as the stop value, start is assumed to be 0 and step is assumed to be 1. If 2 values are given they are assumed to be the start and the stop values and the step is assumed to be 1.

This throws an exception if the range would result in an infinite loop.

[zip [padLeft <expr>] [padRight <expr>] <leftIterator> <rightIterator>]

Return an iterator that produces TwoTuple objects that contain the leftIterator and the rightIterator. The padLeft and padRight options and their associated <expr> are optional and may be emitted. If both options are omitted then the iterator returned by zip terminates when the shorter of the two iterators ends.

If padLeft is given then the value returned by the given <expr> is used to pad results for the leftIterator when it is finished but the rightIterator is not yet exhausted.

If the padRight is given then the converse is true.

Specifying a padLeft and a padRight value will not cause an infinite loop. When both of the input iterators are exhausted, then the zip iteration is done.

The order of the options may be moved around. The last padLeft and <expr> (or padRight and <expr>) is the one that will be used. If a third input iterator is given, it is ignored.

Conditional

[caseList [case …​], [case …​] …​]

The caseList function is built to work with case functions, but this is not necessary. Find will evaluate each of its arguments, in order, until it finds a result. A result is found if the argument either evaluates to true or is an interable object and its first element evaluates to true. In the case of an iterable element, the second element in the iterable is returned as the actual result. In the case of a non-iterable, then just true is returned. If nothing is found, then null (not false) is returned.

Case Lists are slightly preferred over [if] constructs because the implementation of if has to short-circuit the evaluating iterator in order to skip over the then clause without evaluating it. This works, but is less elegant.

[case <predicate> <success>]

The case function is useful when used with the find function. Case takes two arguments and returns a list of two results. The first argument to case is a predicate. If this evaluates to true, then the second argument is evaluated and the list [true, r] is returned where r is the result of the second expression’s evaluation. If the predicate evaluates to false then the list [false, null] is returned.

[defaultCase <success>]

Equivalent to [case [t] [...]].

[if <predicate> <true branch> <false branch>]

If predicate is true, then the true branch is evaluated and returned. If predicate is false and if has not been curried or composed with another function, the true branch is skipped and the false branch is evaluated and returned. If your true branch has no side effects and is not computationally expensive, this should not make any difference.

[isitr <argument>]

Check if the given argument is iterable or not. This also includes types such as iterators which, while they are not "Iterable" in a Java language sense, they are things we may iterate over and something Itrex will iterate over.

[t]

Return true.

[f]

Return false.

[and <arg1> <arg2>…​]

This returns the logical and of the arguments. An argument is considered false if it is literally a False object or null. It is true otherwise. If no arguments are given, this defaults to true.

[or <arg1> <arg2>…​]

This returns the logical and of the arguments. An argument is considered false if it is literally a False object or null. It is true otherwise. If no arguments are given, this defaults to false.

[not <arg>]

Invert and return the logical inversion of the last argument. [not a_string] evaluates to false. If more than 1 argument is given the inversion of the last one is returned.

[eq <args>…​]

Return true if all arguments are Comparables and equal to each other.

[lt <args>…​]

Return true if all arguments are Comparables and are in ascending order.

[lte <args>…​]

Return true if all arguments are Comparables and are in ascending order or adjacent elements are equal.

[gt <args>…​]

Return true if all arguments are Comparables and are in descending order.

[gte <args>…​]

Return true if all arguments are Comparables and are in descending order or adjacent elements are equal.

Variables

[let …​]

Create a child scope. This scope is discarded when the let expression finishes evaluating. Values set with set will then be discarded. The last value passed to let is what is returned.

[get <name>]

Return a value previously set by a call to set or that the user has injected in the EvaluationContext.

[set <name> <value>]

Set the name to the given value. If there is already a value set, it is discarded.

[update <name> <value>]

Update the name to the given value in the context in which is was defined. If there is not already a value set, this is an error and an exception is raised.

Concurrency

Note
 These function will easily crash your program. The core API is not thread-safe. These are provided as a way to safely call your thread safe function implementations, should you choose to write your own functions.
[thread <iterator>]

This takes a single iterator as an argument and wraps it in another iterator which is returned. When an element is fetched from the returned iterator a call to next() on the argument iterator is scheduled and a Future is returned to the caller. Order from the source iterator is no guaranteed. Results from this function may be passed to join to block and unwrap the results.

[join <iterator>]

This takes a single iterator that returns Future+s. The +thread function can map an iterator to an iterator of futures.

[join [thread [my_thread_safe_iterator]]]

Performance

Passing the result of thread directly to join will result in single threaded performance. This is because most functions attempt to only evaluate something if asked for it. As such, nothing is scheduled to be done by thread until join asks for it. Since join blocks for every Future it receives we will never enjoy the parallelism available.

Single Threaded Performance
[list
    [join
        [thread [get "my_threadsafe_iterator"]]]]

One way to improve this to materialize all the Future objects returned by the iterator from thread into a list before passing that list to join.

Threaded Performance with a List
[list
    [join
        [list [thread [get "my_threadsafe_iterator"]]]]]

The downside of this approach is that we must pay the memory cost of a list.

Itrml - A simple expression language

Itrml, pronounced (It-er-am-l), is a particular implementation of a very simple S-Expression language. There is, intentionally, nothing very interesting in it. It is meant to capture and encode data for use by ItrEx when JSON is too verbose.

Data Types

  • Lists - Lists are any sequence of literals or lists surrounded by square ([]) braces. Elements of a list are optionally separated by commas. Commas may be used, or not used, interchangeably in a list. This is to support a very natural function calling syntax of

    [myFunction arg1, arg2, arg3]

    Notice how the arguments are separated by commas, but the function call is not followed by a comma. This is a stylistic choice.

  • Literals - anything that is not a list.

    • Integer valuers. These are any sequence of digits not suffixed with l or L which denote a Long value.

    • Long values. These are denoted by any sequence of only integers followed by nothing, an l or an L. The following are all valid Long integers.

      2341l
34L

    • Double values. They are denoted by a series of digits with an optional decimal point and more digits. A Double may be terminated with a d or D to distinguish it from a Long when there are no decimal digits. For example.

      1.0
32d
3D
4.4

      The above are all double values.

    • Quoted Strings - Any sequence of characters surrounded by ". Characters may be escaped such that the string value abc\"123\" would result in the string value abc"123".

    • Words - Unquoted Strings. These are any token that is not quoted. It is taken to be a string. There are no identifiers or variables in this expression language, just values. Semantic meaning is added by ItrEx if the resultant structure is passed to it for evaluation.

Functions

To define a simple function, you simply call the function command on a thing to be evaluated.

A simple function.
[[function [if [hasArg] [arg] []]] hi]
Naming a function.
[let
    [set f [function [if [hasArg] [arg] []]]]
    [[get f] hi]
]

When a function is called, the evaluation context for it has a special value set, the arguments value. You can access the values in the arguments iterator by calling [hasArgs], [arg] and [args]. See the api documentation for the exact behavior of each.

ItrEx Math Evaluator

[import com.github.basking2.sdsai.itrex.packages.JavaMathPackage]

ItrEx Math Evaluator

ItrEx imports java.util.Math's static functions. Here are some examples.

Example 1: Get the absolute value.
[abs -1]
Example 2: Get the absolute value of a float.
[abs [toFloat -1]]
Example 3: Get the absolute value of which ever is bigger.
[abs [max [toDouble 1] 3d]]
Example 4: Round each value.
[map [curry round] [list 1.1 1.2]]

The preceeding example is a little complex. The curry function is only fetching the round function out to hand it to map. The map call is going to apply round to each value in the list, 1.1 and 1.2.

The result of this expression is an Iterator that returns the Long values 1 and 1.

In the next example we introduce using compose.

Example 5: Complicated
[map
    [compose
        [curry round]
        [curry max 2.3]]
    [list 1.1 1.2 5]]

The compose function will take two functions and pass the output of the last to the input of the other. So, [compose f g 1] is equivalent to calling f(g(1)).

The first use of curry does the simplistic case of just returning the round function. The second use of curry actually does something interesting, it binds the argument 2.3 to max such that a new function, one that only accepts 1 argument, is returned.

The two functions are returned. Each element in the list is then compared with 2.3 and the maximum is returned. That value is then rounded.

The result is an Iterator that returns the values 2, 2 and 5.