TopStreamTutorialDayTwo
Back to TopStreamTutorialDayOne. Forward to TopStreamTutorialDayThree.
Contents
Exercise 3: Luminosity, prescales, and other conditions
Online conditions
When ATLAS starts collecting real data, there will be a network of databases keeping track of online conditions, as depicted here. We will access a single database prototype, the "RunLumiDB," which represents all of these datasets for the streaming test. The creation and use of this database is well-documented at this page. It may also be useful to consult the StreamingTestForUsers TWiki
We have copied the RunLumiDB to SLAC and dCache at BNL. It can also be found at LXPLUS: sqlite://schema=/afs/cern.ch/user/g/gallas/scratch0/mycool.db.
and at SLAC:
sqlite://schema=/afs/slac.stanford.edu/g/atlas/work/a/andr/StreamingTutorial/DB/mycool.db
The code should work with 13.0.10, one needs to check out and compile the packages
-r LumiBlockData-00-00-03 LumiBlock/LumiBlockData -r LumiBlockAthenaPool-00-00-02 LumiBlock/LumiBlockPers/LumiBlockAthenaPool -r LumiBlockTPCnv-00-00-01 LumiBlock/LumiBlockPers/LumiBlockTPCnv -r LumiBlockComps-00-00-08 LumiBlock/LumiBlockComps
You can take a sneak-peek at this database by setting up a release 13 nightly build and running:
LumiCalc.py --help
and then perhaps
LumiCalc.py --database="sqlite://schema=/afs/cern.ch/user/g/gallas/scratch0/mycool.db;dbname=STRMP200" --trigger="L2_e25i" --r=0
Note that the deadtime ( 1.0 - livetime ) is non-negligible. What is the mean livetime fraction of run 1? What is the mean delivered luminosity?
Metadata
Marjorie's talk From Marjorie's talk , you already know the importance of the luminosity block metadata. The inclusive electron stream dataset we prepared for you already has this metadata included in it. This will be the model for ATLAS operations as well -- the metadata will be added to fiels early in the reconstruction chain, and will be updated in the output files when the input files are merged or stripped. However, for realistic release 12 data (i.e. the streaming data), the metadata is not normally present since it was developed for release 13. You can create the metadata store in your AOD using release 13 nightlies, and save it to a standalone file, a ROOT file, or a local copy of your AOD.
Exercise 3.1: Getting the luminosity metadata into the metadata store and writing it
The luminosity blocks and runs for each event are stored in the event header already, so "making the luminosity metadata" is simply the task of putting this information into a simple, ROOT-readable format. This is done in the LumiBlockComps package. It only works in release 13. The luminosity metadata store will only contain the lumiblocks explicitly in your AOD events, if you make it this way. Don't run this tool over a filtered dataset or your cross-sections will be wrong.
[you@lxplus] source ~/cmthome/setup.csh -tag=13.X.0,rel_3,setup [you@lxplus] mkdir $TestArea #if needed [you@lxplus] cd $TestArea
You should now get the file LumiBlock/LumiBlockComps/share/MakeMetaDataNT_jobOptions.py. Depending on your release, get_files might work.
[you@lxplus] emacs MakeMetaDataNT_jobOptions.py; #put your AODs into the EventSelector.InputCollections, change output filename [you@lxplus] athena -l ERROR MakeMetaDataNT_jobOptions.py
Exercise 3.2: Using the luminosity metadata
To use the metadata in a file, just pass the file as the last argument to LumiCalc.py instead of a run or run range. The luminosity blocks in the file will be used for the query instead.
LumiCalc.py --trigger=L2_e25i --database="sqlite://schema=mycool.db;dbname=STRMP200" ../streamtest.004881.inclEle.merge.AOD.v12000605_tid009349._00001.pool.root.1
Cross Section
This is a big task for a tutorial, so you should partner up and divide the work. You have just assembled the tools to
- measure efficiencies directly from data
- measure the luminosity of a sample
and you probably have previous experience in
- estimating backgrounds from Monte Carlo
- calculating the signal acceptance.
(If not, someone next to you can help out!)
Please use the tools below and get as far as you can! *-_-*
Important caveat: Bad luminosity blocks
A small fraction (3%) of the streaming test datasetis missing events from luminosity blocks (but not entire blocks) in one or more streams. The data was lost after the SFO, before merging and reconstruction began. The corresponding luminosity blocks are incomplete (and hence bad for cross-section measurements, since lumiblocks are the smallest useful unit of data for ATLAS bookkeeping.)
The incomplete blocks are easy to spot if you look for the same events in corresponding streams (such as inclusive electrons and exclusive electrons + overlaps.) Since we are only working with one stream today, you should only notice the problem if you can measure a cross section to better than 3% uncertainty. Good luck!
Questions
Why does LumiCalc.py give different luminosity number for the same run, depending on how it is called???
Details from Charles:
Here are the commands I used. The first one is by run number, e.g.
young@yakut03 $ LumiCalc.py --database="sqlite://schema=/afs/slac.stanford.edu/g/atlas/work/a/andr/StreamingTutorial/DB/mycool.db;dbname=STRMP200" --trigger=L2_e25i --r=0
Then I repeated the exercise specifying the files for each run, e.g.
young@yakut03 $ LumiCalc.py --database="sqlite://schema=/afs/slac.stanford.edu/g/atlas/work/a/andr/StreamingTutorial/DB/mycool.db;dbname=STRMP200" --trigger=L2_e25i /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004881.inclEle.merge.AOD.v12000605_tid009349._00001.pool.root.1 /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004881.inclEle.merge.AOD.v12000605_tid009349._00002.pool.root.1 /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004881.inclEle.merge.AOD.v12000605_tid009349._00003.pool.root.1
young@yakut03 $ LumiCalc.py --database="sqlite://schema=/afs/slac.stanford.edu/g/atlas/work/a/andr/StreamingTutorial/DB/mycool.db;dbname=STRMP200" --trigger=L2_e25i /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004892.inclEle.merge.AOD.v12000605_tid009348._00001.pool.root.1 /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004892.inclEle.merge.AOD.v12000605_tid009348._00002.pool.root.1 /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004892.inclEle.merge.AOD.v12000605_tid009348._00003.pool.root.1
Wild cards work for filenames, e.g.
young@yakut03 $ LumiCalc.py --database="sqlite://schema=/afs/slac.stanford.edu/g/atlas/work/a/andr/StreamingTutorial/DB/mycool.db;dbname=STRMP200" --trigger=L2_e25i /afs/slac/g/atlas/work/a/andr/StreamingTutorial/AODs/streamtest.004980.inclEle.merge.AOD.v12000605_tid009352._*.pool.root.* | egrep "Run|IntL"
Results follow. Luminosity obtained by summing over AOD files is the same or less than that obtained querying by run.
Run IntL (by run) IntL (by file) --- ------------- -------------- 0 1817.88 1817.88 1 1602.23 1490.57 2 1503.26 1451.71 3 1499.28 1499.28 4 1778.73 1778.73 5 1430.57 1430.57 6 1488.50 1329.99 7 1293.83 1293.83 8 1532.35 1476.35 9 1516.45 1471.88
The answer (Charles)
We can use --debug to see which LB's are included when doing it by files. Here is an updated table. I checked the numbers in detail for one run (Run 2), and the difference is exactly accounted for by the missing IB.
Run IntL (by run) IntL (by file) Missing IB# --- ------------- -------------- ----------- 0 1817.88 1817.88 none 1 1602.23 1490.57 4, 25 2 1503.26 1451.71 7 3 1499.28 1499.28 none 4 1778.73 1778.73 none 5 1430.57 1430.57 none 6 1488.50 1329.99 11, 16, 17 7 1293.83 1293.83 none 8 1532.35 1476.35 10 9 1516.45 1471.88 19
Are the missing blocks caused by job crashes?
2007-07-20: Prafulla spotted a typo: the last digit in run 1 lumi is 7, not 9.
Luminosity result for our sample
Running on all the AOD files, Charles gets:
IntL (nb^-1) : 15040.77 L1/2/3 accept: 336262 147471 0 Livetime : 15118.7522 Good LBs : 292 BadStatus LBs: 0
Running on all the runs (--rs and --ru):
IntL (nb^-1) : 15463.07 L1/2/3 accept: 345637 151609 0 Livetime : 15530.9247 Good LBs : 300 BadStatus LBs: 0
In both cases, --trigger=L2_e25i. The first number is what should be used for analysis, the second (all runs) is for illustration only.