InfoSphere DataStage Jobs in a Project

Data Stage Jobs in a Project

File systems can have a hard limit on number of subdirectories that can be contained in a single directory. For JFS on AIX they are 32766 (32K-2). Whereas coming to Linux and VXF/HFS on HP-UX, they are EXT3. For each and every job, The Data Stage will create hash (#) files as its subdirectories in the project directory. I will show you the list of the subdirectories that are created for every job. In the below list I have represented the number (or id) of the job as "mo".

• DS_TEMPmo
• RT_CONFIGmo
• RT_STATUSmo
• RT_LOGmo
• RT_BPmo
• RT_BPmo.O
• RT_SCmo (only for parallel jobs)

The number of subdirectories in a directory are limited to 32766, the maximum no.of jobs for a project will be, in theory, approximately 4600 parallel jobs (32766/7 -; 80 where 80 is for overhead of project default directory) or 5380 server jobs (32766/6 - 80). In practice, the number might be smaller due to subdirectories not counted  as above, such as directories created by Quality Stage jobs.

Symptom

When you try to save a newly created job an error will be seen as below, if the max number of subdirectories is reached in the project directory. Error on CREATE.FILE command. Creating file "RT_CONFIGmo as type 30, mkdbfile: cannot create file  RT_CONFIG4817. Unable to create operating system file "RT_CONFIG4817"

Note: “As I said earlier "mo" is the job number of the newly created job”.

Resolving the Problem

• For identifying that if the limit of subdirectories in the project directory has been reached, run the command below. An error will be given in case maximum number has reached:  "mkdir Test"
• If the maximum number of subdirectories is reached, consider any one of the following three solutions.
• Exporting jobs in the project category by category, reallocate jobs into new projects.
• Use EXT4 if IS server is on Linux platform, however, from project management point of view, this needs to be considered carefully as it might be hard to manage a project that contains more than 4,000 jobs;
• Move some subdirectories to another directory then use a softlink to link them back into the project directory. This approach requires lots of system management efforts and is recommended to engage with your systems administrator.