Configuration
Command Line Options
You must specify either the -f or the -js command line option.
| Option | Description |
|---|---|
| -p filename | Specify an alternate name for properties file. The default name is datapump.properties |
| -f filename | Read and execute commands from a script file. Refer to scripts for details. |
| -js jobsetname | Reset and execute all jobs in the specified jobset. |
| -js jobsetname -resume | Resume processing of all jobs in the specified jobset. |
| -cancel | Cancel all executing jobs. |
datapump.properties
This file must be in a CLASSPATH directory or specified on the command line using the -p filename option.
| Property | Required | Default | Description |
|---|---|---|---|
| servicenow.url | Yes | none | URL of the ServiceNow instance |
| servicenow.username | Yes | none | |
| servicenow.password | Yes | none | |
| pumper.sql.url | Yes | none | JDBC URL for the destination database. |
| pumper.sql.username | Yes | none | Username used to connect to the destination database. This user must have CREATE TABLE role. |
| pumper.sql.password | Yes | none | |
| pumper.sql.dialect | Yes | none | References a section in the sqltemplates.xml file. Possible values are oracle1, oracle2 or mysql1. |
| pumper.sql.autocreate | No | true | Indicates whether or not the application should automatically create tables as necessary in the destination database. |
| servicenow.chunksize | No | 100 | Records will be read from ServiceNow 100 rows at a time unless this property is overridden. The maximum permitted value is 250. |
| servicenow.tablename.chunksize | No | none | Allows the chunksize to be overridden for a specific table.
The maximum permitted value is 250.
Performance can be improved by using a larger chunksize for tables with short records. Example: servicenow.cmdb_rel_ci.chunksize=250 |
| servicenow.getkeys.size | No | 10000 | If Use Keys is true (i.e. getRecords is preceded by getKeys to determine the sys_ids of records to be retrieved) then this is the maximum number of keys be be retrieved in each call to getKeys. If there are more keys than this number, then the application will make repeated calls to getKeys until all keys have been retrieved. If this value is less than 1 then the application will attempt to retrieve all keys in single SOAP request. |
| pumper.check_readable | No | true | The Web Services API does not enable the application to distinguish between a ServiceNow table that is empty versus one that is inaccessible for security reasons. If this property is true, then the application will check at startup to verify that at least one record can be read from each table. If a table appears to be empty (i.e. no records can be read) then an exception will be thrown. |
| pumper.check_at_end | No | true | This property is only applicable if Use Keys is set to false for a particular Load operation. In this case the application uses __first_row and __last_row to window the getRecords Web Services calls. If check_at_end is true then the reading will stop as soon as an empty result set is returned. If check_at_end is false then reading will stop as soon as a result set returns fewer than the table chunk size |
| pumper.signal_file | No | none | Used to cleanly terminate the application while it is running. This is the name of a local file. If specified, the file (if it exists) will be removed when application starts and the application will continuously check for the existence of the file. If the application detects the existence of the file it will throw an Interrupted Exception and terminate. |
sqltemplates.xml
This file contains information necessary to generate SQL statements. This file must be in a CLASSPATH directory. Included in this file are
- Properties that control whether column names should upper case or lower case and whether or not they should be in quotes
- Session initialization statements
- Data type mappings that specify the SQL data type for each ServiceNow data type
- Templates for the actual SQL statements (CREATE TABLE, INSERT, UPDATE, etc.)
- Field name mappings
The SQL Templates file is divided into dialects. Three dialects are included.
- oracle1 - Oracle data types. Strings over 4000 characters are generated as CLOBs. Column names are quoted allowing reserved words to be used as column names.
- oracle2 - Column names are not quoted. Instead an underscore is appended to any column name which corresponds to an Oracle reserved word. So, for example, "number" becomes "number_". Strings over 4000 characters are generated as VARCHAR2(4000).
- mysql1 - MySQL data types. Column names are quoted. This dialect has not been extensively tested.
This file should not require modification unless the data types are wrong or the field name mappings need to be modified.
log4j.properties
This file must be in a CLASSPATH directory. It contains logging configuration.
For the syntax of Log4J configuration files please refer to http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PropertyConfigurator.html
The names of loggers in the "soap" package will be prefixed with "ServiceNow". The names of loggers in the "datapump" package will be prefixed with "DataPump". Most loggers for the SOAP package are NOT based on the class name. Instead, there are two loggers for each table, named as follows:
- ServiceNow.Request.tablename
- ServiceNow.Response.tablename
Diagnostic information prior to the SOAP call will be written to the Request logger. Diagnostic information after the SOAP call will be written to the Response logger.
In general, logger levels are as follows
| Level | Description |
|---|---|
| FATAL | Suppress warning messages (not recommended) |
| WARN | Show warning messages |
| INFO | Show one line per SOAP or Database call |
| DEBUG | Show XML sent and received |
| TRACE | Field level diagnostics (these are extremely verbose and should only be used for debugging) |
The following settings are reasonable defaults.
log4j.rootLogger=INFO, C1 log4j.logger.DataPump=INFO log4j.logger.ServiceNow=WARN log4j.logger.ServiceNow.Request=WARN log4j.logger.ServiceNow.Response=WARN # C1 is a ConsoleAppender which outputs to System.out log4j.appender.C1=org.apache.log4j.ConsoleAppender log4j.appender.C1.layout=org.apache.log4j.PatternLayout log4j.appender.C1.layout.ConversionPattern=%p %c %m%n
The following tables are accessed frequently, so high logger levels are recommended for normal processing.
log4j.logger.ServiceNow.Request.u_datapump_jobset=WARN log4j.logger.ServiceNow.Response.u_datapump_jobset=WARN log4j.logger.ServiceNow.Request.u_datapump_job=WARN log4j.logger.ServiceNow.Response.u_datapump_job=WARN log4j.logger.ServiceNow.Request.u_m2m_datapump_jobs=WARN log4j.logger.ServiceNow.Response.u_m2m_datapump_jobs=WARN log4j.logger.ServiceNow.Request.sys_db_object=WARN log4j.logger.ServiceNow.Response.sys_db_object=WARN log4j.logger.ServiceNow.Request.sys_dictionary=WARN log4j.logger.ServiceNow.Response.sys_dictionary=WARN