Derivation Framework Tutorial

The material for this page comes mostly from the standard ATLAS Twiki: XAODMiniTutorialDerivations.

However, this page is self contained for the purpose of adding our own supplementary examples.

Introduction

In this hands-on session you will accomplish the following:

  1. Run the existing derivations and examine the output
  2. Learn how to edit the existing derivations
  3. Create your own derivation using existing tools

The complete derivation framework documentation may be useful as a reference or for some bathroom reading. It can be found at: https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/DerivationFramework

Running the existing derivations

In this first part you'll run the existing test derivations to get a feeling of how things work. Note that we are using a nightly build for this example. This is because the framework is under development and using a nightly build allows us to access the latest changes and fixes. If we used a fixed release, we'd have to check out and compile a bunch of packages to bring things up-to-date.

Setup the environment

In the ATLAS-ready machine of your choice, create a new work directory in a fresh terminal session

setupATLAS
mkdir XAODTutorialDerivations
cd XAODTutorialDerivations
asetup 19.0.3.2,here

Just to make sure everything works, we'll check out specific tags of the framework packages:

cmt co -r DerivationFrameworkCore-00-01-17 PhysicsAnalysis/DerivationFramework/DerivationFrameworkCore
cmt co -r DerivationFrameworkExamples-00-00-18 PhysicsAnalysis/DerivationFramework/DerivationFrameworkExamples

Compile the packages

setupWorkArea.py
cd $TestArea/WorkArea/cmt
cmt br cmt config
cmt br make

The first command above runs a script which generates a top-level requirements file for easier compilation of everything in your working directory. For more handy things like this, take a look at SoftwareTutorialUsefulLinks.

Now create a run directory and a link to a sample file:

mkdir Run
cd Run
ln -s /afs/cern.ch/atlas/project/PAT/xAODs/r5534/valid2.117050.PowhegPythia_P2011C_ttbar.digit.AOD.e2657_s1933_s1964_r5534/AOD.01482225._000107.pool.root.1 xAOD_mc.pool.root

Note: the above path is for lxplus usage, and you should modify it according to the file's location on your machine. If you're working on PDSF or at SLAC, see the site-specific instructions at XAOD Software Tutorial at LBL.

Run the existing derivations

To run the derivations we use a reconstruction transform command:

Reco_tf.py --inputAODFile xAOD_mc.pool.root --outputDAODFile mcTest --reductionConf TEST1 TEST2 TEST3 TEST4 TEST5 TEST6 TEST7

This will run all 7 of the test derivations shown in the presentations and create outputs for each.

  • Once the job has finished, inspect the log file which is called log.AODtoDAOD. Pay particular attention to the event count and overlaps at the end - when assembling a derivation for a group, you'd be expected to check overlaps with other formats before requesting production.
  • Next, take a look at the output of the job - there should be seven pool.root files corresponding to the seven TEST carriages. You will note that only one of them has substantive content, with the rest containing a few KB of metadata and nothing else. Why?

Edit existing derivations

The test derivations are defined in the job options files in PhysicsAnalysis/DerivationFramework/DerivationFrameworkExamples/share/.

  • Take a look through the examples and note the important snippets in each one
    • Tool creation and registration in the tool service
    • Setting up the kernel algorithm. How do the tools get specified to the kernel?
    • Setting up the output stream
  • Try to modify a couple of the derivations
    • Add a new cut to a string-based selection
    • Add some content to the output slimming example

Before running again, you'll need to apply your changes and compile any modified source code (repeat any time you want to apply a change):

cd $TestArea/WorkArea/cmt
cmt br cmt config
cmt br make

Create your own derivation

Now we will setup your own derivation using existing tools. For more advanced requirements it may be necessary to write your own skimming, thinning, or augmenting tool, but given sufficient time most standard use-cases will be handled by tools provided by CP groups. In that case one will only need to create a new job options which sets up the derivation and add a corresponding python object to the DerivationFrameworkProdFlags. We will walk you through these steps below.

The goal of this section is to define a derivation that applies a simple lepton + jets selection.

Writing your (bare) job options

For simplicity we will add our derivation to the existing DerivationFrameworkExamples package.

  • Start by editing a new file under the share directory of this package called DFTutorial.py. Add the following code, and make sure you understand it

#==============================================================================
# Set up common services and job object.
# This should appear in ALL derivation job options
#==============================================================================
from DerivationFrameworkCore.DerivationFrameworkMaster import *
 
#==============================================================================
# Set up your various skimming and augmentation tools here
#==============================================================================

#==============================================================================
# Create the derivation kernel algorithm
#==============================================================================
from DerivationFrameworkCore.DerivationFrameworkCoreConf \
    import DerivationFramework__DerivationKernel
DerivationFrameworkJob += CfgMgr.DerivationFramework__DerivationKernel(
    'DFTUTKernel',
    SkimmingTools = [],
    AugmentationTools = [])

#==============================================================================
# Set up stream
#==============================================================================
streamName = derivationFlags.WriteDAOD_DFTUTStream.StreamName
fileName = buildFileName( derivationFlags.WriteDAOD_DFTUTStream )
DFTUTStream = MSMgr.NewPoolRootStream( streamName, fileName )

# Only events that pass the filters listed are written out
# AcceptAlgs  = logical OR of filters
# RequireAlgs = logical AND of filters
DFTUTStream.AcceptAlgs(['DFTUTKernel'])

#==============================================================================
# Set up slimming content list here
#==============================================================================

Adding the derivation to the prod flags

Before you start adding actual content control to your derivation, make it work in the framework first.

  • In the DerivationFrameworkCore package, open python/DerivationFrameworkProdFlags.py
  • Towards the bottom, add a definition for a WriteDAOD_DFTUTStream JobProperty, and copy the details from one of the other examples.

Make sure to add the new JobProperty to the jobproperties.DerivationFrameworkProdFlags and add the stream name to the listAODtoDPD.

At this point your derivation is not at all interesting, but you could run it with the transform command and get an output.

Adding selections to your derivation

Next, you will use what you've learned to construct a derivation with some specific requirements:

  • Select events that have
    • at least one good electron or muon with pt > 20 GeV and |eta| < 2.5
    • at least three jets with pt > 20 GeV and |eta| < 2.5
  • For the electron quality selection, use the medium selection as demonstrated in TEST3 (StringsToolsExample.py).
  • For the muon quality selection, use the CP tool GoodMuons selection as demonstrated in TEST5 (CPToolsExample.py).
  • Make the final selection using the string-based tool, as demonstrated in both of the above examples
  • Define the slimming to only save the branches we care about: those related to the electrons, muons, jets, and the event info. This is demonstrated in TEST4 (SlimmingExample.py).

Run your derivation

After running the compilation commands, try running your new derivation (DFTUT) using the reconstruction transform. If everything works, then inspect your output. How does the size compare to the test example outputs from before? Open up the output file in an interactive ROOT session and inspect the TTree content using either a TBrowser or printing from the command line. Did you get the output content that you expect from your slimming?

Submitting to the grid

This section is optional. Try submitting your derivation to the grid! Not sure yet if this will work. Needs to be tested

Retrieved from "http://www-atlas.lbl.gov/atlaswiki/index.php?title=Derivation_Framework_Tutorial&oldid=7513"