GEOS–Chem v9–01–02 Online User's Guide

Previous | Next | Printable View (no frames)

7. Coding and Debugging

7.1 Coding: Practice and Style

7.1.1 GEOS–Chem program units

The GEOS–Chem source code is written in Fortran–90 and contains three different types of program units:

1. Traditional subroutines and functions (*.F). These are compiled individually. One subroutine or function is included per file.

2. Header files (*.h). These files are not compiled, but are inlined into other program units via the #include C-preprocessor statement. This is a consistent way of importing global parameters into several subroutines or functions at once. From version 8–02–03, all header files are in the Headers/ subdirectory of the code directory.

   Note: In version 9–01–02, all header and common block files (*.h, CMN*) were replaced with Fortran-90 modules to allow for compatability with ESMF. For more information, see this wiki page.

3. Fortran-90 Modules (*_mod.F). The module construct allows you to bundle several global variables and routines into one logical package, which can then be used by other program units. Also, with modules you may define dynamically allocatable arrays, which are necessary to conserve memory. Much of the newer GEOS–Chem code has been written in module form.

   Note: tpcore_fvdas_mod.F90 is the sole exception to the above-mentioned naming convention, since it ends with a *.F90 extension. This denotes that it is written in free-format Fortran-90 code. Module tpcore_fvdas_mod.F90 is third-party software from S-J Lin at GSFC.

The Makefiles that are provided with GEOS–Chem determine the sequence in which individual program units will be compiled. To allow the make command to determine the order of compilation and to compile the code on multi-processors, each routine is listed in the Makefiles with the module referenced by it (see Chapter 7.1.3 for more information).

7.1.2 Fortran Programming Style

We have attempted to create a clear and consistent programming style for GEOS–Chem subroutines and modules. Some of our style points include:

• Routine header detailing original release date, most recent modification date, list of arguments as input/output, and modification history in ProTeX format.

• References to other variables or routines contained within Fortran-90 modules

• Use of IMPLICIT NONE to force explicit declaration of all variables

• Header files in-lined via the C-preprocessor #include facility

• Comments aligned with mode -- major sections denoted by !=====

• Copious use of white space to enhance readability

• Indentation of IF-THEN-ELSE and DO-ENDDO blocks

• OpenMP parallel DO-loops

When making upgrades to the code, it is good practice to block off the section that you are replacing with comments, and label that as Prior to MM/DD/YY. Then later on, when the upgraded code has been declared stable, you can remove the blocked-off section.

For more information, please be sure to read Appendix 7: GEOS–Chem Style Guide, as well as the floating point and math issues page on the GEOS–Chem wiki.

7.1.3 List of GEOS–Chem Modules

As seen in Chapter 3, the GEOS–Chem distribution is now ordered in different subdirectories. The aim is to separate core routines for GEOS–Chem and 3rd party code. Here is a list of the directories as of GEOS–Chem 9–01–02:

The GEOS–Chem distribution currently contains approximately 40 Fortran-90 modules. As discussed above, modules can be used to include both variables and routines into a single convienient package. A list of the current modules and their dependencies follows.

tpcore_geos5_window_mod.F90

7.1.4 Compilation Sequence for GEOS–Chem

The different subdirectories of the Code directory are compiled in the following order:

KPP/
GeosUtil/
ISOROPIA/
GeosCore/

If you are compiling for a GEOS–Chem mercury simulation with the Global Terrestrial Mercury Model option, then the compilation will proceed in this order:

GeosUtil/
KPP/
GTMM/
ISOROPIA/
GeosCore/

In each subdirectory, there is a Makefile listing all files (functions, subroutines and modules) and the dependencies of each file.

For subroutines, functions or modules which do not refer to other modules, only the file itself is listed in the dependencies, e.g.

decomp.o                     : decomp.F

When there are references to other modules in the file (through USE statement), you need to list the file itself and all other modules from the same directory it refers to. E.g: from the table in List of GEOS–Chem Modules, tropopause_mod.F refers to 9 other modules but only 3 are in GeosCore, so in the Makefile you write:

tropopause_mod.o             : tropopause_mod.F          \
                               comode_mod.o   dao_mod.o  \
                               logical_mod.o

The order of the list of files is not important. The make utility defines the order of compilation by itself. Thus we maintain an alphabetical order of the files.

7.1.5 OpenMP parallel loop directives

In the late 1990's, a new open standard for parallel computing named OpenMP was developed by several compiler vendors, including Sun, SGI, Compaq, and Microsoft. The resulting standard allows parallel processing source code (written in either Fortran or C) to be ported between different platforms with minimal effort. As of version 4.17, all GEOS–Chem parallel DO loops have been converted to the new OpenMP standard.

Here is an example DO loop written using OpenMP parallel directives:

Column   1         2         3         4
1234567890123456789012345678901234567890
!$OMP PARALLEL DO
!$OMP+SHARED( A )
!$OMP+PRIVATE( I, J )
!$OMP+SCHEDULE( DYNAMIC )
      DO J = 1, JJPAR
      DO I = 1, IIPAR
	     A(I,J) = A(I,J) * 2.0
      ENDDO
      ENDDO
!$OMP END PARALLEL DO

You may also have noticed that the first column in each line of the !$OMP PARALLEL DO directives begin with a legal Fortran comment character (either C or !). This is by design. In order to invoke the parallel procesing commands, you must turn on a specific switch in your makefile (this is -mp for SGI; check your compiler manual for other platforms). If you do not specify multiprocessor compilation, then the parallel processing directives will be considered as Fortran comments, and the associated DO-loops will be executed on one processor only. Also, since GEOS–Chem uses the traditional "fixed-form" Fortran style, the !$OMP commands must begin at column 1. Otherwise a syntax error will result at compile time.

It should be noted that OpenMP commands are not the same as MPI (message passing interface). With OpenMP directives, you are able to split a job among several processors on the same machine. You are NOT able to split a job among several processors on different machines. Therefore, OpenMP is not suitable for Beowulf-style architectures. We are working on porting GEOS–Chem to distributed style architectures.

For more information about OpenMP, consult the web site OpenMP.org.

7.2 GEOS–Chem Debugging

GEOS–Chem is an evolving model. New features and functionalities are constantly being added to it by a rapidly increasing group of users. As with any software project, mistakes are inevitable.

Most of the bugs you you will encounter when working on GEOS–Chem will fall into one of two categories:

1. True bugs: typos, omissions, reading the wrong file, or other outright mistakes.

2. Design limitation bugs: that is, writing code in such a way that will make it difficult or impossible to extend the functionality of the model at some future time.

Bugs of the first category are, in general, easily rectified. The fixes to these bugs typically involve either correcting a misspelled word or updating an incorrect numerical value.

Bugs belonging to the second category can be rather pernicious. In almost all instances, you will find that the model is working fine -- that is, until you try to add in a new chemistry simulation, diagnostic, or third-party routine. Then you may find that a major modification to the structure of GEOS–Chem is necessary before the new code can be successfully interfaced.

GEOS–Chem is a combination of several different indivdual pieces: emissions, chemistry and deposition routines from Harvard, transport and convection routines from GSFC, photolysis from UC Irvine, etc. Therefore, the structure of GEOS–Chem was (and still is) largely defined by the structure of the individual pieces from which it was created. It is not always possible to deviate from this set structure without having to rewrite entire sections of source code.

7.2.1 Recent fixes

Between each GEOS–Chem release, we strive to fix bugs as we find them.

• For the most recent fixes, look at the Bugs and Fixes page on the GEOS–Chem wiki. It is the most up-to-date document for bugs. For each of them, you will find a description, how to fix it and which version of GEOS–Chem includes the fix.

• We shall update the Common GEOS–Chem Error Messages wiki page with a list of bugs that commonly occur. Many of these bugs can be fixed by providing the proper input to GEOS–Chem, or by recompiling with the proper options.

If you are developing code for GEOS–Chem, then please view the following resources:

1. Appendix 7: GEOS–Chem Style Guide
2. GEOS–Chem wiki. Floating point math issues
3. GEOS–Chem wiki: Machine issues & portability

These pages may help you resolve some common issues.

7.2.2 Debugging Tips

Here are some steps you can take to try to diagnose a particular GEOS–Chem error.

1.	Turn on diagnostic ND70 in the file input.geos. This will cause debugging messages (via routine debug_msg contained in error_mod.F) to be written to the log file after operations such as transport, chemistry, emissions, dry deposition, etc. are called from the main program. In this way, you should be able to identify in which operation the error occurred. NOTE: debug_msg will cause the text to be flushed to disk after it is printed. Most Unix systems feature buffered I/O; that is, the contents of a file or screen output are not updated until an internal buffer (usually 16K of memory) is filled up. If you don't flush the error message to disk, then the last output to the log file may not accurately indicate the location at which the error occurred. Therefore, we recommend using debug_msg instead of a standard Fortran WRITE or PRINT* statement.
2.	It may also be necessary to include additional debug statements into main.F or other routines. This may be done by calling subroutine debug_msg as follows CALL DEBUG_MSG( '### after routine X' ) By adding several of these debug statements, you should be able to track down the particular place at which the error is occurring.
3.	If you want to check for negative concentrations, NaN or Infinite values, you can add the following line of code: CALL CHECK_STT('Where I think there is a problem') Make sure to add the following line at the beginning of the subroutine where you call CHECK_STT: USE TRACER_MOD,  ONLY: CHECK_STT
4.	If you want to just print out the minimum and maximum value of an array variable that is not included in met_field_debug, then simply add the following line of code: PRINT*, '### Min, Max: ', MINVAL( X ), MAXVAL( X ) CALL FLUSH( 6 ) where X is the name of the variable. If you want to print out the sum of X instead of the min and max, add this line of code: PRINT*, '### Sum of X : ', SUM( X ) CALL FLUSH( 6 ) Here we also have to call the Fortran routine FLUSH(6), which ensures that the output (in this case, to the screen, unit #6), will be immediately written to disk.
5.	It is a good idea to periodically compile your code with array-out-of-bounds error checking. This will make sure that all of the arrays are being accessed with indices whose values fall within the specified array dimensions. For example, if you have the following situation in your code: REAL*8 :: A(10), B(10) ... DO I = 1, 10 B(I) = A(I+1) ENDDO then the code will try to access the 11th element of the A array. But since A only has 10 elements, the code will try to access the next contiguous memory location, which may belong to a different variable altogether. Therefore, a "junk" value will be copied into the 10th element of the B array. This may result in an error message such as: forrtl: severe (174): SIGSEGV, segmentation fault occurred Sometimes an array-out-of-bounds error will mask itself as a segmentation fault. The above error message is from the Intel Fortran Compiler; other compilers will give similar error messages. To invoke array-out-of-bounds checking in your code, make clean, and then make sure you compile with the following Makefile options: make realclean make -j4 BOUNDS=yes Intel Fortran Compiler users may also add the TRACEBACK=yes, which will print out a list of all routines that were being called at the time of the error. Then recompile your code as usual. Any errors will be detected at runtime, and you should get error output such as: forrtl: severe (408): fort: (3): Subscript #1 of the array PBL_THICK has value -1 which is less than the lower bound of 1 The above error was generated with the Intel Fortran Compiler. Other compilers should give similar errors (although the error numbers may differ). NOTE: Once you have located and fixed the offending array statement, you should recompile to make an executable without the array-out-of-bounds checking built in. The error checking is thorough, but it can cause your code to slow down noticeably. Therefore, it is only recommended to use array-out-of-bounds checking in debugging runs and not in production runs.

7.2.3 Using a debugger

If you have access to a debugger, you should spend a little bit of time learning how to use it, at least the basic usage. Knowing how to:

• examine data when a program stops
• navigate the stack when a program stops
• set breakpoints

will save you a lot of time when debugging a program.

To use a debugger, you typically need to compile your code with a flag that tells the compiler to generate symbolic debug information. This is usually done with -g. You also need to switch off any optimization flag, since they can "rewrite" your code and you may not be able to access all variables. For example, GEOS–Chem is compiled with the following flags on IFORT when using a debugger:

FFLAGS = -cpp -w -noalign -convert big_endian -g

instead of:

FFLAGS = -cpp -w -O2 -auto -noalign -convert big_endian

To switch to the debug options simply add DEBUG=yes (case sensitive) to the make command.

NOTE: Compiling with the debug option does not turn off the parallelization. Some debugger do not accept parallelized code, to turn off the parallelization, add OMP=no (case sensitive) to the make command.

Previous | Next | Printable View (no frames)