CDF Production Tutorial

Instructions

Quick Start
1) Log-in on the appropriate machine.

old location: cdfprd@fncdfsrv0.fnal.gov
new location: cdfprd@fcdflnx4(Doesn't work!)

kinit USERNAME_aris
ssh cdfprd@fncdfsrv0.fnal.gov

2) Create/update configuration files


cd ~/ProdConfig

______________________________________________________________________________
Configuration file structure: 
farm.ProductionExe_$pass_$stream

For example: stream B, pass 0m

farm.ProductionExe_0m_bphysr

Inside configuration file there are families of parameters for different jobs.

Project.*
DatasetCreator.*
CronSubmit.*
MergeSubmit.*
SamStore.*
_______________________________________________________________________________


3) Create output directories on concatenation servers

List of concatenation servers:
fncdfsrv15,16,17,18,31,32,33,34,35
 
i) Login to project server (i.e.fncdfsrv31) 
ssh cdfprd@fncdfsrv13.fnal.gov
cd /export/stage/ 
 
ii) Run scripts
$ ~/testRel/production_6.1.4/cdfprd/gen/init_directories ProductionExe_0m_bphysr 0m bphysr


(Change 0m and bphysr depending on the corresponding pass and stream)

4) Update Metalia bookkeeping (Create Metalia directories for each stream)

For example, for stream B:

cdfprd@fncdfsrv0:~/Projects]$
mkdir  ProductionExe_0k_bphysr 
cd ProductionExe_0k_bphysr
mkdir Metalia
mkdir logs

Metalia bookkeeping files keep the track of processed files. The names of production output files can be derived from production input files, for each output file SAM metadata are created upon the output file is copied to the concatenation server. There is a cron job for each production projects that periodically updates Metalia files according to the database records about output files. CronSubmit? script that submits production jobs submits only jobs for input files that have unprocessed files in the Metalia file. Metalia files are created using DatasetCreator? script.



5) Run DatasetCreator 
    cdfprd@fncdfsrv0:~/testRel/production_6.1.4]$ source_me
$ ./cdfprd/DatasetCreator/datasetCreator.sh -x THE_PROJECT=ProductionExe_0m_bphysr -x DOIT=1

$ ./cdfprd/DatasetCreator/datasetCreator.sh -x THE_PROJECT=ProductionExe_0m_cphysr -x DOIT=1

$ ./cdfprd/DatasetCreator/datasetCreator.sh -x THE_PROJECT=ProductionExe_0m_ephysr -x DOIT=1

$ ./cdfprd/DatasetCreator/datasetCreator.sh -x THE_PROJECT=ProductionExe_0m_gphysr -x DOIT=1

$ ./cdfprd/DatasetCreator/datasetCreator.sh -x THE_PROJECT=ProductionExe_0m_hphysr -x DOIT=1

$ ./cdfprd/DatasetCreator/datasetCreator.sh -x THE_PROJECT=ProductionExe_0m_jphysr -x DOIT=1

    • The result of DatasetCreator? is a configuration file: 
    Example:
/home/cdfprd/Projects/ProductionExe_0m_bphysr/Metalia/ProductionExe_0m_bphysr.conf
    The bookkeeping files are created upon the first submission of production job, one file for run/run range in the same directory.



    Clean up after production period


    • before clean up be sure that severe error summary is updated (instructions)
    • Clean up past production log files from output directories on concatenation servers. It is best to delete directories after backing up log files. tar directories and move them to cdfprd@fncdfsrv0:/cdf/proj/127.prod_log/Archive










      • ssh into the production concatenation server
      • Stop all cronjobs for the project (production/concatenation/upload) 
    HOW TO EDIT CRONTAB EXAMPLE:
ssh -l cdfprd fncdfsrv9
crontab -e
     
    Remember, crontab is responsibility of a user and is not backed up. If unsure of yourself, save it to some file: 
    crontab -l >& my_crontab_copy.txt


    • cd /export/stage/
    • cd into the project directory
    • tar up the logs, crashed and recovery directories,
    • tar file name should contain stream name, period number, server name
    • check that resulting tar file is meaninful and copy it to archive directory 

    EXAMPLE
ssh -l cdfprd fncdfsrv9
cd /export/stage
cd ProductionExe_0m_bphysr_P26
tar -czvf bphysr_P26_srv9.tgz ./crashed/* ./logs/* ./recovery/* 
scp bphysr_P26_srv9.tgz cdfprd@fncdfsrv0:/cdf/proj/127.prod_log/Archive/bphysr_P26_srv9.tgz



      • remove the project directroy, example: "rm -rf ProductionExe_0j_gphysr/"

    Prepare for the next production period


    Create new output directories with latest production pass. Use script






















    • For example, for stream G (fncdfsrv13.fnal.gov)
    • cd /export/stage/
    • ~/testRel/production_6.1.4/cdfprd/gen/init_directories ProductionExe_0j_gphysr 0j gphysr
    • check that directory and all sub directories of /export/stage/ProductionExe_0j_gphysr are present.


    • Edit each project configuration page ( ~/ProdConfig/farm.$ProjectName )

    • Include latest production tarball by placing it in the directory /home/cdfprd/Tarballs/, example: /home/cdfprd/Tarballs/ProductionExe_6.1.1c_f_p11_maxopt.tar.gz

    • Create Metalia directories for output of DatasetCreator.


    Start running jobs


    1) Edit crontab on fncdfsrv0.

    crontab -e
# create your entry
     

    About

    Search

    PISIKA Copyright © 2009