GEOS–Chem's Makefiles determine how to compile individual program unit. Each routine (and the modules that it references) are listed in makefiles. This lets the GNU Make utility determine the order of compilation. Refer to Chapter 7.1.3 for more information.

7.1.2 Fortran Programming Style

We have attempted to create a clear and consistent programming style for GEOS–Chem subroutines and modules. Some of our style points include:

• A routine header list of arguments and modification history (in ProTeX format)

• References to other variables or routines contained within modules

• Use of IMPLICIT NONE to force explicit declaration of all variables

• Comments aligned with mode -- major sections denoted by !=====

• Copious use of white space to enhance readability

• Indentation of IF-THEN-ELSE and DO-ENDDO blocks

• OpenMP parallel DO-loops

• NEW: Passing data between subroutines with Fortran derived type objects (We are doing this to facilitate connecting GEOS–Chem to the NASA GEOS–5 GCM.)

When making upgrades to the code, block off the section that you are replacing with comments, and label that with Prior to MM/DD/YY. You can remove the blocked off section once your modified code has been declared stable.

For more information, please be sure to read Appendix 7: GEOS–Chem Style Guide, as well as the floating point and math issues page on the GEOS–Chem wiki.

7.1.3 List of GEOS–Chem Modules

As described in Chapter 3, we have separated the GEOS–Chem source code files into various subdirectories. We sought to separate core GEOS–Chem routines from 3rd-party code.

Here we list the directory structure in GEOS–Chem v9–02:

The GEOS–Chem distribution currently contains over 150 module files. As discussed above, we employ modules to package both variables, subroutines, and functions into a single logical program unit. We list all of the modules and their dependencies below:

tpcore_geos5_window_mod.F90

tpcore_geosfp_window_mod.F90

7.1.4 Compilation Sequence for GEOS–Chem

GEOS–Chem's makefiles compile source code files in the following order:

1. Files in NcdfUtil/
2. Files in Headers
3. Files in KPP/
4. Files in GeosUtil/
5. Files in ISOROPIA/
6. Files in GeosCore/

If you select the GEOS–Chem mercury simulation with the Global Terrestrial Mercury Model option, then the makefiles will compile files in this order::

1. Files in GTMM/
2. Files in NcdfUtil/
3. Files in Headers
4. Files in KPP/
5. Files in GeosUtil/
6. Files in ISOROPIA/
7. Files in GeosCore/

Each directory has its own makefile. The makefile contains a list of all of the source code files in the subdirectory, plus dependent routines (a.k.a. the "dependencies" list). Source code files that do not refer to any modules have a dependencies listsuch as:

decomp.o                     : decomp.F

Whereas a source code file that refers to several modules (in this case, GeosCore/tropopause_mod.F) has a dependencies list like this:

tropopause_mod.o             : tropopause_mod.F                     \
                               comode_mod.o         diag_mod.o      \
                               logical_mod.o

Looking at the List of GEOS–Chem Modules, you will see that GeosCore/tropopause_mod.F refers to 13 other modules. But 10 of those modules are found in different directories (such as Headers/ and GeosUtil/), which are compiled prior to GeosCore/. Therefore you can ignore these when creating the dependencies listing. In the example above, we have only specified the 3 modules that are located in the same directory as tropopause_mod.F.

The order of modules in the dependencies list does not matter. The GNU Make utility will define the order of compilation from the list of files (and their dependencies) that you have specified in each Makefile. You will not have to worry about this unless you add new source code files. If you have questions about how to add dependency lists to makefiles, please let us know.

7.1.5 OpenMP parallel loop directives

In the late 1990's, several compiler vendors defined a new open standard for parallel computing, aptly named OpenMP. Most modern Fortran and C compilers include support for OpenMP, which facilitates porting parallel source code to different platforms. GEOS–Chem has used OpenMP parallel computing since version 4.17.

GEOS–Chem uses OpenMP parallelization to gain computational advantage by using several CPUs. Recall that most GEOS–Chem arrays represent quantities (i.e. meteorological fields, tracer concentrations, chemical species concentrations) on a geospatial grid. Therefore, OpenMP parallelization essentially assigns each CPU its own region of the "world" to work on. While each CPU is restricted to performing computations on its own region of the "world", it can see the entire "world"—that is, all of the memory on the system. This is illustrated by the figure below.

It is important to remember that OpenMP parallelization is a loop-level parallelization. Only commands within selected DO loops will execute in parallel. All GEOS–Chem simulations begin executing on a single CPU (known as the "master CPU"). Upon entering a parallel DO loop, other CPUs will be invoked to share the workload within the loop. At the end of the parallel DO loop, the other CPUs return to standby status and the execution continues only on the "master" CPU.

Here is an example of the OpenMP commands that you need to add to a GEOS–Chem DO loop in order to make it execute in parallel:

Column   1         2         3         4
1234567890123456789012345678901234567890
!$OMP PARALLEL DO
!$OMP+SHARED( A )
!$OMP+PRIVATE( I, J )
!$OMP+SCHEDULE( DYNAMIC )
      DO J = 1, JJPAR
      DO I = 1, IIPAR
	     A(I,J) = A(I,J) * 2.0
      ENDDO
      ENDDO
!$OMP END PARALLEL DO

Notice that the first column of each !$OMP directive has a Fortran comment character (!). This is by design. In order to invoke OpenMP's parallel directives, you must toggle a specific switch in your makefile (usually -mp or -openmp, depending on your compiler). Otherwise, the compiler will interpret the OpenMP parallel directives as Fortran comments. This would cause all OpenMP parallel DO-loops revert to regular Fortran DO loops, executing on a single CPU.

Because the above example uses the traditional fixed-format (Fortran–77) style, the !$OMP directives must begin at column 1. Otherwise a syntax error will result at compile time. If you are using the free-format (Fortran–90) style, then you may indent the !OMP directives.

OpenMP parallelization is not equivalent to MPI (message passing interface) parallelization. With OpenMP, you can execute parallel code across CPUs of the same machine or cluster node. You CANNOT use CPUs from physically separated machines or nodes. Therefore, OpenMP is incompatible with distributed architectures (e.g. Beowulf, grid computing, cloud computing). Our grid-independent GEOS–Chem project seeks to remove this bottleneck by making make GEOS–Chem compatible with MPI parallelization.

For more information about OpenMP, consult the web site OpenMP.org.

7.2 GEOS–Chem Debugging

The GEOS–Chem model evolves constantly. Our rapidly-increasing user base frequently submits updates and new features for inclusion into the standard GEOS–Chem repository. As with any software project, mistakes are inevitable.

The bugs you will encounter when working on GEOS–Chem fall into one of two categories:

1. True bugs: typos, omissions, reading the wrong file, or other outright mistakes.

2. Design limitation bugs: that is, writing code in such a way that precludes future expansion.

You can easily rectify the bugs of the first category. Fixing these usually means correcting a misspelled word, updating an incorrect numerical value, reading from the proper file, etc.

You may struggle to correct the bugs of the second category. You'll find that your simulation works perfectly—until you try to add in a new species, reaction, offline chemistry simulation, diagnostic, or third-party code package. You may have to invest significant time and effort in modifying GEOS–Chem before it finally does what you want..

Remember that GEOS–Chem includes several disparate software elements: emissions, chemistry and deposition routines from Harvard, transport and convection routines from GSFC, photolysis from UC Irvine, etc. The structure of these codes have largely dictated the overall structure of GEOS–Chem. At present, we are working to rectify many of these legacy-code issues (a.k.a. "historical baggage).

7.2.1 Recent fixes

We strive to fix bugs in GEOS–Chem as soon as we find them. Please see our Bugs and Fixes page on the GEOS–Chem wiki. On this page, you will find a list of bugs by GEOSChem version. Clicking on a link will take you to a wiki post which explains the bug (and how it was fixed) in detail.

When your GEOS–Chem simulation dies with an error, first refer to our Common GEOS–Chem Error Messages wiki page. You may find a solution there.

We invite you to peruse these useful resources:

1. Appendix 7: GEOS–Chem Style Guide
2. GEOS–Chem wiki. Floating point math issues
3. GEOS–Chem wiki: Machine issues & portability

7.2.2 Debugging with the GEOS–Chem Unit Tester

The GEOS–Chem Unit Tester is a package of scripts and Makefiles that will compile and run several short (~1 model hour) GEOS–Chem simulations with a set of standard debugging flags. Each simulation is designed to uncover several common bugs for different combinations of meterology, horizontal grid, and simulation type. Using the Unit Tester helps us to ensure that source code updates from developers do not introduce any unforeseen errors into future GEOS–Chem versions.

You can download the GEOS–Chem Unit Tester with Git:

git clone git://git.as.harvard.edu/bmy/GEOS-Chem-UnitTest

The GEOS–Chem Unit Tester package has several subdirectories:

doc/ jobs/ logs/ perl/ runs/

In the perl/ directory you will find the main program (gcUnitTest) and a sample input file (UnitTest.input). You may edit the input file to specify which GEOS–Chem test simulations (i.e. combinations of meteorology, grid, and simulation type) you would like the Unit Tester to run. For example, adding these lines to the input file:

#---------|-----------|------|----------------|------------|------------|
#  MET    | GRID      | NEST | SIMULATION     | START DATE | END DATE   |
#---------|-----------|------|----------------|------------|------------|
  geos5     2x25        -      fullchem        2013070100   2013070101
  geosfp    4x5         -      fullchem        2005070100   2005070101
  geos5     4x5         -      RnPbBe          2005070100   2005070101
  merra     4x5         -      soa             2005070100   2005070101

will tell the GEOS–Chem Unit Tester to schedule the following tests: