Loading...

InfoSphere DataStage Jobs in a Project

Data Stage Jobs in a Project

File systems can have a hard limit on number of subdirectories that can be contained in a single directory. For JFS on AIX they are 32766 (32K-2). Whereas coming to Linux and VXF/HFS on HP-UX, they are EXT3. For each and every job, The Data Stage will create hash (#) files as its subdirectories in the project directory. I will show you the list of the subdirectories that are created for every job. In the below list I have represented the number (or id) of the job as "mo".
  • DS_TEMPmo
  • RT_CONFIGmo
  • RT_STATUSmo
  • RT_LOGmo
  • RT_BPmo
  • RT_BPmo.O
  • RT_SCmo (only for parallel jobs)












The number of subdirectories in a directory are limited to 32766, the maximum no.of jobs for a project will be, in theory, approximately 4600 parallel jobs (32766/7 -; 80 where 80 is for overhead of project default directory) or 5380 server jobs (32766/6 - 80). In practice, the number might be smaller due to subdirectories not counted  as above, such as directories created by Quality Stage jobs.

Symptom

When you try to save a newly created job an error will be seen as below, if the max number of subdirectories is reached in the project directory. Error on CREATE.FILE command. Creating file "RT_CONFIGmo as type 30, mkdbfile: cannot create file  RT_CONFIG4817. Unable to create operating system file "RT_CONFIG4817"

Note: “As I said earlier "mo" is the job number of the newly created job”.

Resolving the Problem

  • For identifying that if the limit of subdirectories in the project directory has been reached, run the command below. An error will be given in case maximum number has reached:  "mkdir Test"
  • If the maximum number of subdirectories is reached, consider any one of the following three solutions.
  • Exporting jobs in the project category by category, reallocate jobs into new projects.
  • Use EXT4 if IS server is on Linux platform, however, from project management point of view, this needs to be considered carefully as it might be hard to manage a project that contains more than 4,000 jobs;
  • Move some subdirectories to another directory then use a softlink to link them back into the project directory. This approach requires lots of system management efforts and is recommended to engage with your systems administrator.


Training 6595310144090779791

Post a Comment

emo-but-icon

Home item

Blog Archive

Popular Posts

Random Posts

Flickr Photo