GEOS–Chem's Makefiles determine how to compile individual program unit. Each routine (and the modules that it references) are listed in makefiles. This lets the GNU Make utility determine the order of compilation. Refer to Chapter 7.1.3 for more information.

7.1.2 Fortran Programming Style

We have attempted to create a clear and consistent programming style for GEOS–Chem subroutines and modules. Some of our style points include:

• A routine header list of arguments and modification history (in ProTeX format)

• References to other variables or routines contained within modules

• Use of IMPLICIT NONE to force explicit declaration of all variables

• Header files in-lined via the C-preprocessor #include facility

• Comments aligned with mode -- major sections denoted by !=====

• Copious use of white space to enhance readability

• Indentation of IF-THEN-ELSE and DO-ENDDO blocks

• OpenMP parallel DO-loops

• NEW: Passing data between subroutines with Fortran derived type objects (We are doing this to facilitate connecting GEOS–Chem to the NASA GEOS–5 GCM.)

When making upgrades to the code, block off the section that you are replacing with comments, and label that with Prior to MM/DD/YY. You can remove the blocked off section once your modified code has been declared stable.

For more information, please be sure to read Appendix 7: GEOS–Chem Style Guide, as well as the floating point and math issues page on the GEOS–Chem wiki.

7.1.3 List of GEOS–Chem Modules

As described in Chapter 3, we have separated the GEOS–Chem source code files into various subdirectories. We sought to separate core GEOS–Chem routines from 3rd-party code.

Here we list GEOS–Chem 9–01–03's directory structure:

The GEOS–Chem distribution currently contains over 150 module files. As discussed above, we employ modules to package both variables, subroutines, and functions into a single logical program unit. We list all of the modules and their dependencies below:

tpcore_geos5_window_mod.F90

tpcore_geos57_window_mod.F90

7.1.4 Compilation Sequence for GEOS–Chem

GEOS–Chem's makefiles compile source code files in the following order:

1. Files in NcdfUtil/
2. Files in Headers
3. Files in KPP/
4. Files in GeosUtil/
5. Files in ISOROPIA/
6. Files in GeosCore/

If you select the GEOS–Chem mercury simulation with the Global Terrestrial Mercury Model option, then the makefiles will compile files in this order::

1. Files in GTMM/
2. Files in NcdfUtil/
3. Files in Headers
4. Files in KPP/
5. Files in GeosUtil/
6. Files in ISOROPIA/
7. Files in GeosCore/

Each directory has its own makefile. The makefile contains a list of all of the source code files in the subdirectory, plus dependent routines (a.k.a. the "dependencies" list). Source code files that do not refer to any modules have a dependencies listsuch as:

decomp.o                     : decomp.F

Whereas a source code file that refers to several modules (in this case, GeosCore/tropopause_mod.F) has a dependencies list like this:

tropopause_mod.o             : tropopause_mod.F                     \
                               comode_mod.o         dao_mod.o       \
                               diag_mod.o           logical_mod.o

Looking at the List of GEOS–Chem Modules, you will see that GeosCore/tropopause_mod.F refers to 9 other modules. But 5 of those modules are found in different directories (such as Headers/ and GeosUtil/), which are compiled prior to GeosCore/. Therefore you can ignore these when creating the dependencies listing. In the example above, we have only specified the 4 modules that are located in the same directory as tropopause_mod.F.

The order of modules in the dependencies list does not matter. The GNU Make utility will define the order of compilation from the list of files (and their dependencies) that you have specified in each Makefile. You will not have to worry about this unless you add new source code files. If you have questions about how to add dependency lists to makefiles, please let us know.

7.1.5 OpenMP parallel loop directives

In the late 1990's, several compiler vendors defined a new open standard for parallel computing, aptly named OpenMP. Most modern Fortran and C compilers include support for OpenMP, which facilitates porting parallel source code to different platforms. GEOS–Chem has used OpenMP parallel computing since version 4.17.

Here is a sample OpenMP parallel DO loop:

Column   1         2         3         4
1234567890123456789012345678901234567890
!$OMP PARALLEL DO
!$OMP+SHARED( A )
!$OMP+PRIVATE( I, J )
!$OMP+SCHEDULE( DYNAMIC )
      DO J = 1, JJPAR
      DO I = 1, IIPAR
	     A(I,J) = A(I,J) * 2.0
      ENDDO
      ENDDO
!$OMP END PARALLEL DO

Notice that the first column of each !$OMP directive has a Fortran comment character (!). This is by design. In order to invoke OpenMP's parallel directives, you must toggle a specific switch in your makefile (usually -mp or -openmp, depending on your compiler). Otherwise, the compiler will interpret the OpenMP parallel directives as Fortran comments. This would cause all OpenMP parallel DO-loops revert to regular Fortran DO loops, executing on a single CPU.

Because the above example uses the traditional fixed-format (Fortran–77) style, the !$OMP directives must begin at column 1. Otherwise a syntax error will result at compile time. If you are using the free-format (Fortran–90) style, then you may indent the !OMP directives.

OpenMP parallelization is not equivalent to MPI (message passing interface) parallelization. With OpenMP, you can execute parallel code across CPUs of the same machine or cluster node. You CANNOT use CPUs from physically separated machines. Therefore, OpenMP is incompatible with distributed architectures (e.g. Beowulf, grid computing, cloud computing). Our grid-independent GEOS–Chem project seeks to remove this bottleneck by making make GEOS–Chem compatible with MPI parallelization.

For more information about OpenMP, consult the web site OpenMP.org.

7.2 GEOS–Chem Debugging

The GEOS–Chem model evovles constantly. Our rapidly-increasing user base frequently submits updates and new features for inclusion into the standard GEOS–Chem repository. As with any software project, mistakes are inevitable.

The bugs you will encounter when working on GEOS–Chem fall into one of two categories:

1. True bugs: typos, omissions, reading the wrong file, or other outright mistakes.

2. Design limitation bugs: that is, writing code in such a way that precludes future expansion.

You can easily rectify the bugs of the first category. Fixing these usually means correcting a misspelled word, updating an incorrect numerical value, reading from the proper file, etc.

You may struggle to correct the bugs of the second category. You'll find that your simulation works perfectly—until you try to add in a new species, reaction, offline chemistry simulation, diagnostic, or third-party code package. You may have to invest significant time and effort in modifying GEOS–Chem before it finally does what you want..

Remember that GEOS–Chem includes several disparate software elements: emissions, chemistry and deposition routines from Harvard, transport and convection routines from GSFC, photolysis from UC Irvine, etc. The structure of these codes have largely dicated the overall structure of GEOSChem. At present, we are working to rectify many of these legacy-code issues (a.k.a. "historical baggage.)

7.2.1 Recent fixes

We strive to fix bugs in GEOS–Chem as soon as we find them. Please see our Bugs and Fixes page on the GEOS–Chem wiki. On this page, you will find a list of bugs by GEOSChem version. Clicking on a link will take you to a wiki post which explains the bug (and how it was fixed) in detail.

When your GEOS–Chem simulation dies with an error, first refer to our Common GEOS–Chem Error Messages wiki page. You may find a solution there.

We invite you to peruse these useful resources:

1. Appendix 7: GEOS–Chem Style Guide
2. GEOS–Chem wiki. Floating point math issues
3. GEOS–Chem wiki: Machine issues & portability

7.2.2 Debugging Tips

Here are some steps you can take to try to diagnose a particular GEOS–Chem error.

1.	Turn on diagnostic ND70 in the file input.geos. ND70 tells GEOS–Chem to call subroutine DEBUG_MSG from various locations in the code (after advection, chemistry, emissions, dry deposition, etc). DEBUG_MSG writes a short message to the screen and immediately flushes the output to the log file. This will help you be able to pin down where the error occurred. NOTE: DEBUG_MSG prints and then immediately flushes the error message to disk. Keep in mind that most Unix systems feature buffered I/O—the screen does not refresh until an internal buffer (usually some multiple of 16KB) fills up with text. If you don't flush an error message to disk, then the last log file output may not reflect the location of the error.
2.	Add more calls to DEBUG_MSG as necessary: CALL DEBUG_MSG( '### after routine X' ) CALL DEBUG_MSG( '### after function Y' )
3.	Check for negative concentrations, NaN or Infinite values, call routine CHECK_STT: ! Put this at the top of the subroutine where you are calling CHECK_STT USE TRACER_MOD, ONLY: CHECK_STT . . . ! Check for NaN, infinity, negatives: CALL CHECK_STT( 'Where I think there is a problem' )
4.	Check the minimum and maximum values of an array with the MINVAL and MAXVAL intrinsic functions: PRINT*, '### Min, Max: ', MINVAL( ARRAY ), MAXVAL( ARRAY ) CALL FLUSH( 6 ) Check the sum of an array with the SUM intrinsic function: PRINT*, '### Sum of X : ', SUM( ARRAY ) CALL FLUSH( 6 ) Don't forget to add a CALL FLUSH(6)to flush the output to the screen or log file.
5.	Compile your code with array-out-of-bounds error checking. This makes sure that you are not accessing an array with an invalid index. Consider this example: REAL*8 :: A(10), B(10) ... DO I = 1, 10 B(I) = A(I+1) ENDDO The code will try access the 11th element of the A array. But since A only has 10 elements, the code will access the next contiguous memory location, which belongs to a different variable altogether. This will copy a "junk" value into the 10th element of the B array. Out-of-bounds errors may also manifest themselves as a segmentation fault: forrtl: severe (174): SIGSEGV, segmentation fault occurred To invoke array-out-of-bounds checking in your code, issue these commands: make realclean make -j4 BOUNDS=yes (If you use the Intel Fortran Compiler, you can also add the TRACEBACK=yes, which will print out a list of all routines that were being called at the time of the error.) The compiler will trap out-of-bounds errors at runtime. You may get an error message such as: forrtl: severe (408): fort: (3): Subscript #1 of the array PBL_THICK has value -1 which is less than the lower bound of 1 NOTE: Once you have located and fixed the offending array statement, you should recompile without the array-out-of-bounds checking built in. The error checking causes your code to slow down noticeably. Check for array-out-of-errors in your debugging runs but not in your production runs.

7.2.3 Using a debugger

If you have access to a debugger (IDB, DBX, Totalview), then learn how to use at least the basic commands, such as:

• Examinng data when a program stops
• Navigating the stack when a program stops
• Setting breakpoints

This will save you a lot of time and hassle.

When you use a debugger, you must compile your code with a flag that tells the compiler to generate symbolic debug information. On most compilers, this flag is named -g. You also need to switch off any compiler optimizations (which can modify the sequence in which individual instructions occur) by specifying the -O0 flag.

Add DEBUG=yes (case sensitive) to the make command to apply these options:

make -j4 DEBUG=yes OMP=no    # Without parallelization
make -j4 DEBUG=yes           # With parallelization

Previous | Next | Printable View (no frames)