Skip to content

Oozie Sqoop action

Oozie Sqoop action

There are few ways to build Oozie Sqooping action. You can decide to use:

  • arg properties
  • command property
  • or options file

Each of the options is equally good so you can choose which one suits you best.

Contents

HOW TO

1. arg properties

You can specify all Sqoop options using the arg properties. They go in pairs: one for option preceded by “‐‐” and one for its value, or alone for option which don’t require a value (also with “‐‐”). The only difference is the schema option, which requires 3 args: “‐‐”, “‐‐schema” and “schema_value”. schema option needs to be specified as the last one, otherwise some other parameters coming afterwards will not be visible.
 
Here’s how such Sqoop action may look like:

  • export to database
  • import from database

Note: Instead of “‐‐table” and “‐‐ ‐‐schema” options you may use “‐‐query” option like:

where $CONDITIONS clause in needed for Sqoop to run properly. You can add other filter clauses to where as well.
“query” is only available for import action.

  • call stored procedure
 

2. command property

The alternative to many arg properties is a command property, which allows you to define all Sqoop options in a way that you would run it using Sqoop shell.
Almost all used options require a preceding “‐‐”. The only difference is the schema option, which requires “‐‐ ‐‐”. schema option needs to be specified as the last one, otherwise some other parameters coming afterwards will not be visible.
 
Here’s how such Sqoop action may look like:

  • export to database
  • import from database

Note: Instead of “‐‐table” and “‐‐ ‐‐schema” options you may use “‐‐query” option like:

where $CONDITIONS clause in needed for Sqoop to run properly. You can add other filter clauses to where as well.
“query” is only available for import action.

  • call stored procedure
 

3. options file

Instead of defining all options in the workflow itself, you may define them in the file stored in HDFS. It gives a nice option of switching files without changing the workflow or creating a generic one. Created file needs to be pointed to with “‐‐options-file” option and attached with workflow file property.
Remember to use “‐‐ ‐‐schema” option at the very end.
 
Here’s how such Sqoop action may look like:

and options file:

  • export to database
  • import from database
  • call stored procedure

Leave a Reply

Your email address will not be published. Required fields are marked *