Skip to content

Conditions in the Built-in Audits dont support macro vars #2399

Closed

Description

@erindru

Slack context: https://tobiko-data.slack.com/archives/C044BRE5W4S/p1712179233739539

Built-in audits (such as unique_values) have a condition property that can be used to pass extra conditions like so:

@MODEL (
   ...
  audits (
    unique_values(columns=(id), condition=foo='bar')
  )
)

However, if you want to use @start_dt and @end_dt in the condition to limit the amount of data scanned on the underlying engine, they dont expand out correctly.

For example, this condition fails:

condition=ts between @start_dt and @end_dt

-- ValueError: Unsupported expression type MacroVar

You can wrap it in @SQL to get further:

condition=@SQL('ts between @start_dt and @end_dt')

-- sqlglot.errors.ParseError: Invalid expression / Unexpected token. Line 1, Col: 38.
  ts between 2023-03-01 00:00:00+00:00 and 2023-03-10 23:59:59.999999+00:00
Error: Failed to resolve macro for expression. Error trying to eval macro.

Generated code: self.parse_one(self.template('ts between @start_dt and @end_dt', locals()))

Original sql: @SQL('ts between @start_dt and @end_dt') at '.'

It seems that it doesn't take the current dialect into account and just does a string substitution. That same expression in a normal model would correctly expand to CAST('@start_dt' AS TIMESTAMP) AND CAST('@end_dt' AS TIMESTAMP)

You can work around the dialect issue by writing SQL to convert the string values to timestamps:

condition=@SQL('ts between cast(''@start_dt'' as timestamp) and cast(''@end_dt'' as timestamp)')

However, that's a bit hacky. It would be good if the rules for the model SQL dialect were applied

Activity

  1. self-assigned this
    on Apr 5, 2024
  2. added a commit that references this issue on Apr 5, 2024
    b1c39b4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions