System crash and crashed jobs

When you stop the Lobster Integration Server regularly, the system completes any jobs that are still running. The stop is delayed until these jobs finish. After you trigger the stop, the system no longer starts new jobs.

In some cases, a profile runs for a very long time. This can happen when processing takes an unusually long time or when the target system stops responding in Phase 6. In this state, you cannot restart the Integration Server. The administrator must then decide whether to force the stop. The navigation path is Administration > Base settings > System, DMZ, Loadbalancing.

IMPORTANT:

Before you force the stop, check that no relevant processing jobs are still running. Only long-running or stalled cron operations may remain active. Otherwise, the system may lose data currently being processed.

Do not start the Integration Server again until the old process has fully stopped.

IMPORTANT:

Start the Integration Server only after the old process has fully stopped. Starting it too early can cause an inconsistent system state. Wait briefly after the process ends, because network connections can remain active at the system level for a few seconds.

For more information, see Configure shutdown behavior.

Crashed jobs

See also: Retention periods of backup files, logs, crashed jobs.

If a job does not finish because of a forced stop, the system marks it as a "crashed job" — a job whose processing was interrupted. When you restart Lobster Integration, the system restarts all crashed jobs. Processing does not resume where it stopped; it starts again from the beginning.

If the stop happens after Phase 3, the system has already run all functions. This includes the autonumber functions and database operations within functions. When implementing profiles, developers and users must therefore account for the possibility of a crash and assess the impact of re-running the process.

A crashed job can also result from a computer crash, a hardware failure, or a power outage. Additionally, the system restarts the crashed jobs subsequent to a restart in these instances.

Lobster Integration handles crashed jobs as follows:

• Requirement: In configuration file ./etc/startup.xml, the parameter trackCrashedJobs is set to true.

• Lobster Integration creates a file in the directory ./datawizard/backup/lock each time a job starts.

• After a job finishes, the system deletes the file again, regardless of the outcome.

• The file is named <job number>.lck and you can open it with a text editor.

• It contains the job number, the name of the input file, and the ID (not the name) of the associated profile.

• Because the file does not contain the original payload, the job's backup file must still exist to restart the job.

• If crashed jobs exist, the backups still exist, and the feature is enabled, the system restarts the jobs after an adjustable time (parameter restoreWaitTime in ./etc/startup.xml).

• The detail logs in the Control Center note the restart of a crashed job.

• If the feature is not enabled, the system only creates an entry in the database table dw_log_sum so the crashed job stays visible to users.

In a production system, stabilize the system environment to minimize the frequency of crashed jobs.

Preventing the restart of crashed jobs

Use the parameter startCrashedJobs in configuration file ./etc/startup.xml to prevent the restart of all crashed jobs.

If you want to prevent the restart of individual crashed jobs, search the subdirectory lock of the backup directory for files named <job number>.lck. Delete the relevant files selectively. The default value for the backup directory is ./datawizard/backup. See parameter backupDir in configuration file ./etc/startup.xml.

Use the parameter ignoreOldCrashedJobs to prevent the restart of crashed jobs older than a specified number of days.