Big Data Analytics Jobs

Defines the connection profile for the job.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Spaces

• Variable Name: %%AZURE-ACCOUNT

Determines the ID of the job created in your Databricks workspace.

Defines task parameters to override when the job runs, according to the Databricks convention. The list of parameters must begin with the name of the parameter type. For example:

• "notebook_params":{"param1":"val1", "param2":"val2"}

• "jar_params": ["param1", "param2"]

For more information about the parameter types, review the properties of RunParameters in the OpenAPI specification provided through the Azure Databricks documentation.

For no parameters, specify the following value:

• "params": {}

(Optional) Defines a token to use to rerun job runs that timed out in Databricks.

Values:

• Control-M-Idem_%%ORDERID — With this token, upon rerun, Control-M invokes the monitoring of the existing job run in Databricks. Default.

• Any other value — Replaces the Control-M idempotency token. When you rerun a job using a different token, Databricks creates a new job run with a new unique run ID.

Defines the name of a connection profile to use to connect to the Azure HDInsight workspace.

Determines which parameters are passed to the Apache Spark Application during job execution, in JSON format (name:value pairs).

This JSON must include the file and className elements.

Determines the number of seconds to wait before the Apache Spark batch job is verified.

Default: 10 seconds

Determines which connection profile to use to connect to the Databricks workspace.

Determines the job ID created in your Databricks workspace.

Defines task parameters to override when the job runs, according to the Databricks convention. The list of parameters must begin with the name of the parameter type. For example:

• "notebook_params":{"param1":"val1", "param2":"val2"}

• "jar_params": ["param1", "param2"]

For more information about the parameter types, review the properties of RunParameters in the OpenAPI specification provided through the Azure Databricks documentation.

For no parameters, specify the following value:

• "params": {}

(Optional) Defines a token to use to rerun job runs that timed out in Databricks.

Values:

• Control-M-Idem_%%ORDERID — With this token, upon rerun, Control-M invokes the monitoring of the existing job run in Databricks. Default.

• Any other value — Replaces the Control-M idempotency token. When you rerun a job using a different token, Databricks creates a new job run with a new unique run ID.

Defines the connection profile for the job.

Rules:

• Characters: 1−30

• Case Sensitive: Yes

• Invalid Characters: Spaces

Determines the database that the job uses.

Determines the schema that the job uses.

A schema is an organizational model that describes the layout and definition of fields and tables, and their relationships to each other, in a database.

Determines one of the following Snowflake actions to perform:

• SQL Statement: Runs any number of Snowflake-supported SQL statements, such as queries, calling or creating procedures, database maintenance tasks, and creating and editing tables.

• Copy from Query: Copies a queried database and schema into an existing or new file in cloud storage.

• Copy from Table: Copies from an existing table.

• Create Table and Query: Creates a table, populated by a query, in the specified database and schema.

• Create Snowpipe: Creates a Snowpipe and saves it to a file in cloud storage.

• Start or Pause Snowpipe: Starts or pauses an existing Snowpipe.

• Stored Procedure: Calls an existing procedure and its arguments.

• Snowpipe Load Status: Monitors the status of a Snowpipe for a set period of time.

Determines one or more Snowflake-supported SQL commands.

Rule: Must be written in a single line, with strings separated by one space only.

Determines the maximum number of seconds to run the job in Snowflake.

Determines whether the following job-defining attributes are displayed:

• Parameters

• Role

• Bindings

• Warehouse

Defines Snowflake-provided parameters that let you control how data is presented.

{ 
  "param1":"value1",
  "param2":"value2"
}

Determines the Snowflake role used for this Snowflake job.

A role is an entity that can be assigned privileges on secure objects. You can be assigned one or more roles from a limited selection.

Defines the values to bind to the variables used in the Snowflake job, in JSON format.

For more information on bindings, see the Snowflake documentation.

"1": { 
      "type": "FIXED", 
      "value": "123" 
    } 
"2": { 
      "type": "TEXT", 
      "value": "String" 
    }

Determines the warehouse used in the Snowflake job.

A warehouse is a cluster of virtual machines that processes a Snowflake job.

Determines whether to show a full JSON response in the log output.

Determines the number of seconds to wait before checking the status of the job.

Default: 20

Defines the cloud storage location.

Defines the query used for copying the data.

Defines the storage integration object.

Determines whether to overwrite an existing file in the cloud storage, as follows:

• Yes

• No

Determines one of the following file formats for the saved file:

• JSON

• CSV

Defines where the JSON or CSV file is saved.

You can save to Amazon Web Services, Google Cloud Platform, or Microsoft Azure.

Defines the name of the copied table.

Defines the name of the new or existing table where the data is queried.

Defines the query used for the copied data.

Defines the name of the Snowpipe.

A Snowpipe loads data from files when they are ready, or staged.

Defines the table that the data is copied into.

Defines the stage from where the data is copied.

Determines whether to start or pause the Snowpipe, as follows:

• Start Snowpipe

• Pause Snowpipe

Defines the name of the stored procedure.

Defines the value of the argument in the stored procedure.

Defines the table that is monitored when loaded by the Snowpipe.

Defines the cloud storage location.

A stage is a pointer that indicates where data is stored, or staged.

Determines the number of days to monitor the Snowpipe load status.

Defines the cloud storage location where a CSV file log is created.

The CSV file log details the load status for each Snowpipe.