GEOS–CHEM v6–02–05
User's Guide
Contact: Bob Yantosca (bmy@io.harvard.edu)
The GEOS–CHEM source code is written in Fortran–90 and contains three different types of program units:
1. Traditional subroutines and functions (*.f). These are compiled individually. One subroutine or function is included per file. 2. Header files (*.h, CMN*). These files are not compiled, but are inlined into other program units via the #include C-preprocessor statement. This is a consistent way of importing global common blocks and parameters into several subroutines or functions at once. 3. Fortran-90 Modules (*_mod.f). The module construct allows you to bundle several global variables and routines into one logical package, which can then be used by other program units. Also, with modules you may define dynamically allocatable arrays, which are necessary to conserve memory. Much of the newer GEOS–CHEM code has been written in module form.
Note: tpcore_fvdas_mod.f90 is the sole exception to the above-mentioned naming convention, since it ends with a *.f90 extension. This denotes that it is written in free-format Fortran-90 code. Module tpcore_fvdas_mod.f90 is third-party software from S-J Lin at GSFC.
The Makefile (cf. Section 2.5: Compiling the GEOS–CHEM source code) that is provided with GEOS–CHEM determines the sequence in which individual program units will be compiled. The current order of compilation is as follows:
1. Fortran-90 modules 2. Subroutines and functions (except FAST-J, SLOW-J) 3. Subroutines pertaining to FAST-J photolysis code (if necessary) 4 Subroutines pertaining to SLOW-J photolysis code (if necessary) 5 Subroutines for Bryan Duncan's CO-OH parameterization (if necessary)
In addition, the Fortran–90 modules must themselves be compiled in the proper order. For example, if you have a Module B that makes reference to a Module A, then Module A must be compiled before Module B is compiled. Otherwise a compile-time error will result. To see the current compilation sequence for GEOS–CHEM modules, visit Section 3.3.
We have attempted to create a clear and consistent programming style for GEOS–CHEM subroutines and modules. Some of our style points include:
When making upgrades to the code, it is good practice to block off the section that you are replacing with comments, and label that as Prior to MM/DD/YY. Then later on, when the upgraded code has been declared stable, you can remove the blocked-off section.
For more information, please be sure to read the GEOS–CHEM style guide.
The GEOS–CHEM
distribution currently contains approximately 40 Fortran-90 modules. As discussed
above, modules can be used to include both variables and routines into a single
convienient package. A list of the current modules and their dependencies follows.
|
Module
Name
|
Purpose
|
Other
Modules Referenced
|
|---|---|---|
| Kr85_mod.f | Contains variables and routines used for the 85Kr simulation |
diag_mod.f |
| RnPbBe_mod.f | Contains variables and routines used for the 222Rn-210Pb-7Be-10Be simulation |
diag_mod.f |
| a3_read_mod.f | Contains routines which unzip, open and read the GEOS-CHEM A-3 fields from disk | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f time_mod.f transfer_mod.f |
| a6_read_mod.f | Contains routines which unzip, open and read the GEOS-CHEM A-6 fields from disk | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f time_mod.f transfer_mod.f |
| acetone_mod.f | Contains subroutines to emit the biogenic flux of acetone into the full chemistry simulation | bpch2_mod.f diag_mod.f dao_mod.f error_mod.f grid_mod.f time_mod.f transfer_mod.f |
| aircraft_nox_mod.f | Contains arrays and routines to compute aircraft NOx, emissions for the full chemistry simulation | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f grid_mod.f pressure_mod.f |
| biofuel_mod.f | Contains arrays and routines to compute monthly biofuel burning emissions for NOx, CO, ALK4, ACET, MEK, ALD2, PRPE, C3H8, CH2O, C2H6 CH4, and CH3I | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f tracerid_mod.f transfer_mod.f |
| biomass_mod.f | Contains arrays and routines to compute monthly biomass burning emissions for NOx, CO, ALK4, ACET, MEK, ALD2, PRPE, C3H8, CH2O, C2H6 CH4, and CH3I | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f grid_mod.f time_mod.f tracerid_mod.f transfer_mod.f |
| bpch2_mod.f | Contains the routines used to read data from and write data to binary punch file format (v. 2.0) | error_mod.f file_mod.f julday_mod.f |
| c2h6_mod.f | Contains the variables and routines used to perform a standalone simulation for ethane | bpch2_mod.f biomass_mod.f biofuel_mod.f dao_mod.f diag_mod.f geia_mod.f global_oh_mod.f grid_mod.f time_mod.f tracerid_mod.f transfer_mod.f |
| carbon_mod.f | Contains variables and routines for carbon aerosol emissions and chemistry |
bpch2_mod.f |
| ch3i_mod.f | Contains the variables and routines used to perform a standalone simulation for methyl iodide | bpch2_mod.f biomass_mod.f biofuel_mod.f dao_mod.f diag_mod.f error_mod.f grid_mod.f time_mod.f tracerid_mod.f transfer_mod.f uvalbedo_mod.f |
| charpak_mod.f | Contains selected routines from the CHARPAK package for string manipulation | none |
| chemistry_mod.f | Contains a driver program which selects the proper chemistry routine for the various GEOS–CHEM simulation types |
acetone_mod.f |
| comode_mod.f | Contains allocatable arrays for SMVGEAR that were previously contained in common blocks in header file "comode.h" | error_mod.f |
| convection_mod.f | Contains routines which select the proper convection code for GEOS–1, GEOS–STRAT, GEOS–3 or GEOS–4 met data sets | dao_mod.f diag_mod.f fvdas_convect_mod.f grid_mod.f pressure_mod.f time_mod.f wetscav_mod.f |
| dao_mod.f | Contains arrays that hold DAO met fields, as well as subroutines that compute, interpolate, or otherwise process DAO met field data | error_mod.f grid_mod.f time_mod.f pressure_mod.f |
| diag51_mod.f | Contains variables and routines to generate save timeseries data over the United States where the local time is between two user-defined limits | bpch2_mod.f dao_mod.f error_mod.f file_mod.f grid_mod.f pressure_mod.f time_mod.f tracerid_mod.f |
| diag_mod.f | Contains declarations for allocatable arrays for use with GEOS–CHEM diagnostics and other routines | none |
| drydep_mod.f | Contains variables and routines to compute dry deposition velocities and frequencies | comode_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f grid_mod.f pressure_mod.f time_mod.f tracerid_mod.f |
| dust_dead_mod.f | Contains routines and variables from Charlie Zender's DEAD dust mobilization model. Most routines are from Charlie Zender, but have been modified and/or cleaned up for inclusion into GEOS-CHEM. |
bpch2_mod.f |
| dust_mod.f | Contains routines for computing dust aerosol emissions, chemistry, and optical depths. |
bpch2_mod.f |
| emissions_mod.f | Contains a driver program which selects the proper emissions subroutine for the various GEOS–CHEM simulation types. | c2h6_mod.f carbon_mod.f ch3i_mod.f dust_mod.f error_mod.f global_ch4_mod.f Kr85_mod.f RnPbBe_mod.f seasalt_mod.f sulfate_mod.f tagged_co_mod.f |
| error_mod.f | Contains error checking subroutines | none |
| file_mod.f | Contains file unit numbers and I/O error trapping routines | error_mod.f |
| fvdas_convect_mod.f |
Contains routines (originally from NCAR) which perform shallow and deep convection for the GEOS–4 met fields. These routines account for shallow and deep convection, updrafts, and downdrafts. |
diag_mod.f grid_mod.f pressure_mod.f |
| geia_mod.f | Contains routines used to read and scale the GEIA/Piccot fossil fuel emissions for NOx, CO, and hydrocarbons | bpch2_mod.f file_mod.f grid_mod.f time_mod.f |
| global_ch4_mod.f | Contains variables and routines for simulating CH4 chemistry in the troposphere | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f grid_mod.f pressure_mod.f time_mod.f transfer_mod.f |
| global_hno3_mod.f | Contains variables and routines for reading in monthly mean HNO3 concentrations for the offline sulfate simulation | bpch2_mod.f dao_mod.f error_mod.f transfer_mod.f |
| global_no3_mod.f | Contains variables and routines for reading in global monthly mean NO3 concentrations (used for the offline sulfate simulation) | bpch2_mod.f error_mod.f transfer_mod.f |
| global_nox_mod.f | Contains variables and routines for reading the global monthly mean NOx concentration from disk (used for the offline sulfate simulation) | bpch2_mod.f error_mod.f transfer_mod.f |
| global_oh_mod.f | Contains variables and routines for reading the global monthly mean OH concentration from disk | bpch2_mod.f error_mod.f transfer_mod.f |
| grid_mod.f | Contains variables and routines for computing grid box longitudes, latitudes, and surface areas | error_mod.f |
| gwet_read_mod.f | Contains routines which unzip, open and read the GEOS-CHEM GWET fields from disk. |
bpch2_mod.f |
| i6_read_mod.f | Contains routines which unzip, open and read the GEOS-CHEM I-6 fields from disk | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f time_mod.f transfer_mod.f |
| julday_mod.f | Contains routines for converting YYYYMMDD and HHMMSS to/from astronomical Julian date. (This allows GEOS–CHEM to handle transitions such as end-of-month, end-of-year, and end-of-century in a consistent fashion.) | none |
| lightning_nox_mod.f | Contains variables and routines for emitting NOx from lightning into the atmosphere. The original code comes from the old GISS-II CTM's of Yuhang Wang, Gerry Gardner, & Larry Horowitz. Cleaned up for inclusion into GEOS-CHEM. | dao_mod.f diag_mod.f error_mod.f file_mod.f grid_mod.f pressure_mod.f |
| optdepth_mod.f | Contains routines to compute optical depths for GEOS–1, GEOS–STRAT, GEOS–2, and GEOS–3 met data sets | diag_mod.f |
| phis_read_mod.f | Contains routines which unzip, open and read the GEOS-CHEM PHIS fields (geopotential heights) from disk | bpch2_mod.f dao_mod.f diag_mod.f error_mod.f file_mod.f time_mod.f transfer_mod.f |
| pjc_pfix_mod.f | Contains variables and routines which implement the Philip Cameron-Smith pressure fixer for the new GEOS–4/TPCORE transport code (contained in tpcore_fvdas_mod.f90). | error_mod.f grid_mod.f pressure_mod.f |
| planeflight_mod.f | Contains variables and routines which are used to "fly" a plane through the GEOS–CHEM model simulation. This is useful for comparing model results with aircraft observations. | bpch2_mod.f comode_mod.f dao_mod.f error_mod.f file_mod.f time_mod.f pressure_mod.f |
| pressure_mod.f | Contains variables and routines which specify the grid box pressures for both hybrid or pure-sigma models. This is necessary for running GEOS–CHEM with the new fvDAS meteorology. (dsa, bmy, 8/27/02) | error_mod.f |
| restart_mod.f | Contains variables and routines which are used to read and write GEOS–CHEM restart files, which contain tracer concentrations in [v/v] mixing ratio. | bpch2_mod.f charpak_mod.f dao_mod.f error_mod.f file_mod.f grid_mod.f time_mod.f |
| rpmares_mod.f | Contains variables and routines to compute aerosol thermodynamic equilibrium | dao_mod.f error_mod.f global_hno3_mod.f time_mod.f tracerid_mod.f |
| seasalt_mod.f | Contains
arrays and routines for performing either a coupled chemistry/aerosol run
or an offline seasalt aerosol simulation. The original code taken from Mian
Chin's GOCART model and was modified accordingly for GEOS–CHEM. |
dao_mod.f |
| sulfate_mod.f | Contains the sulfate chemistry routines from Mian Chin's GOCART model |
bpch2_mod.f |
| tagged_co_mod.f | Contains variables and routines used for the geographically tagged CO simulation | biofuel_mod.f biomass_mod.f bpch2_mod.f dao_mod.f diag_mod.f error_mod.f geia_mod.f grid_mod.f global_oh_mod.f global_nox_mod.f global_ch4_mod.f time_mod.f pressure_mod.f transfer_mod.f |
| tagged_ox_mod.f | Contains variables and routines to perform a tagged Ox simulation. P(Ox) and L(Ox) rates need to be archived from a full chemistry simulation before you can run w/ Tagged Ox. |
bpch2_mod.f |
| time_mod.f | Contains routines for computing date & time variables | charpak_mod.f error_mod.f grid_mod.f julday_mod.f |
| toms_mod.f | Contains variables and routines for reading the EP-TOMS O3 column data from disk (for use w/ the FAST-J photolysis routines) | bpch2_mod.f error_mod.f time_mod.f transfer_mod.f |
| tpcore_bc_mod.f | Contains variables and routines to read and write boundary conditions from a global simulation. These are needed for TPCORE transport for nested grid 1 x 1 simulations. | bpch2_mod.f error_mod.f file_mod.f grid_mod.f time_mod.f |
| tpcore_fvdas_mod.f90 | Contains the TPCORE transport subroutine package by S-J Lin, for GEOS–4/fvDAS met fields. This new TPCORE can parallelize under both the OpenMP and MPI standards. | none |
| tpcore_mod.f | Contains the TPCORE transport subroutine package by S-J Lin, version 7.1. Also contains the pressure fixer routines for TPCORE from M. Prather (in versions 4.21 and higher). | dao_mod.f diag_mod.f global_ch4_mod.f grid_mod.f pressure_mod.f time_mod.f |
| tpcore_window_mod.f | Contains the TPCORE transport subroutine package by S-J Lin, version 7.1, modified for nested-grid simulations by Yuxuan Wang. | dao_mod.f diag_mod.f grid_mod.f global_ch4_mod.f grid_mod.f pressure_mod.f time_mod.f |
| tracerid_mod.f | Contains variables and routines used to identify tracers, emission species, biomass burning species, and biofuel burning species | charpak_mod.f error_mod.f |
| transfer_mod.f | Contains routines to compute cast data from REAL*4 to REAL*8 precision, and also to regrid GEOS–3 data from 48 vertical levels to 30 vertical levels. | error_mod.f grid_mod.f pressure_mod.f |
| transport_mod.f | Contains routines which select the proper transport routines for GEOS–CHEM global and window simulations. |
dao_mod.f |
| upbdflx_mod.f | Contains subroutines which impose stratospheric boundary conditions on O3 and NOy | bpch2_mod.f dao_mod.f error_mod.f pressure_mod.f time_mod.f tagged_ox_mod.f tracerid_mod.f transfer_mod.f |
| uvalbedo_mod.f | Contains variables and routines for reading the UV Albedo data from disk (for use w/ the FAST-J photolysis routines). | bpch2_mod.f error_mod.f transfer_mod.f |
| wetscav_mod.f | Contains arrays for used in the wet scavenging of tracer in cloud updrafts, rainout, and washout | dao_mod.f diag_mod.f error_mod.f pressure_mod.f time_mod.f tracerid_mod.f |
3.4 Compilation Sequence for GEOS–CHEM F90 modules
In Fortran-90, modules
must be compiled in the proper order. If Module B references Module A,
then Module A must be compiled before Module B is compiled, otherwise a compile-time
error will result. Bob
Yantosca has determined the order in which GEOS–CHEM modules must be compiled.
Make sure that the following code fragment is listed in your Makefile:
error_mod.o \
julday_mod.o \
file_mod.o \
grid_mod.o \
time_mod.o \
bpch2_mod.o \
pressure_mod.o \
transfer_mod.o \
tracerid_mod.o \
comode_mod.o \
diag_mod.o \
dao_mod.o \
drydep_mod.o \
geia_mod.o \
global_ch4_mod.o \
global_hno3_mod.o \
global_no3_mod.o \
global_nox_mod.o \
global_oh_mod.o \
uvalbedo_mod.o \
RnPbBe_mod.o \
Kr85_mod.o \
acetone_mod.o \
aircraft_nox_mod.o \
biofuel_mod.o \
biomass_mod.o \
c2h6_mod.o \
carbon_mod.o \
ch3i_mod.o \
a3_read_mod.o \
a6_read_mod.o \
i6_read_mod.o \
phis_read_mod.o \
gwet_read_mod.o \
diag51_mod.o \
lightning_nox_mod.o \
optdepth_mod.o \
planeflight_mod.o \
restart_mod.o \
rpmares_mod.o \
wetscav_mod.o \
sulfate_mod.o \
tagged_co_mod.o \
tagged_ox_mod.o \
fvdas_convect_mod.o \
convection_mod.o \
pjc_pfix_mod.o \
dust_dead_mod.o \
dust_mod.o \
seasalt_mod.o \
toms_mod.o \
tpcore_bc_mod.o \
tpcore_fvdas_mod.o \
tpcore_mod.o \
tpcore_window_mod.o \
transport_mod.o \
upbdflx_mod.o \
chemistry_mod.o \
emissions_mod.o
The Unix make utility will compile each module in the order it is listed above. This will ensure that GEOS–CHEM is compiled correctly.
If you should add another module to the GEOS–CHEM source code, you must also make sure that it is compiled after any other modules it may reference.
3.5 OpenMP parallel loop directives
If you are a veteran GEOS–CHEM user, you may remember seeing some parallel DO-loops in various subroutines which looked similar to this:
Column 1 2 3 4
1234567890123456789012345678901234567890
C$DOACROSS SHARE( A )
C$& LOCAL( I, J )
C$& MP_SCHEDTYPE=DYNAMIC
DO J = 1, JJPAR
DO I = 1, IIPAR
A(I,J) = A(I,J) * 2.0
ENDDO
ENDDO
The above fragment of code instructs GEOS–CHEM to split up the multiplication of the A array by 2 among several processors (as was specified by the MP_SET_NUMTHREADS environment variable). Each processor receives a specific array element A(I,J) to work with -- different processors get different elements. Therefore, each processor needs to have its own copies of I and J. In other words, I and J must be declared LOCAL for the parallelization. The entire A array, on the other hand, can remain available for access by all the processors, since the elements of A will be modified one at a time. Therefore, A must be declared SHAREd for the parallelization. The scheduling type in the above example is DYNAMIC. This means that each processor will grab a new element of A as soon as it is finished with the current one. This results in a non-sequential processing of the array elements.
The major drawback of the C$DOACROSS parallel loop commands is that this syntax is only supported on the SGI platform. It is not possible to port GEOS–CHEM to other platforms using C$DOACROSS parallel loops.
In the late 1990's, a new open standard for parallel computing named OpenMP was developed by several compiler vendors, including Sun, SGI, Compaq, and Microsoft. The resulting standard allows parallel processing source code (written in either Fortran or C) to be ported between different platforms with minimal effort. As of version 4.17, all GEOS–CHEM parallel DO loops have been converted to the new OpenMP standard.
Here is the same DO loop as in the above example written using OpenMP parallel directives:
Column 1 2 3 4
1234567890123456789012345678901234567890
!$OMP PARALLEL DO
!$OMP+SHARED( A )
!$OMP+PRIVATE( I, J )
!$OMP+SCHEDULE( DYNAMIC )
DO J = 1, JJPAR
DO I = 1, IIPAR
A(I,J) = A(I,J) * 2.0
ENDDO
ENDDO
!$OMP END PARALLEL DO
The idea
is the same as before. The syntactic differences are:
1. C$DOACROSS is now written as !$OMP PARALLEL DO 2. SHARE( A ) is now written as SHARED( A ) 3. LOCAL( I, J ) is now written as PRIVATE( I, J ) 4. MP_SCHEDTYPE=DYNAMIC is now written as SCHEDULE( DYNAMIC ) 5. The continuation character C$& is now written as !OMP+ 6. You may add an optional !$OMP END PARALLEL DO at the end of the loop
You may also have noticed that the first column in each line of the C$DOACROSS and !$OMP PARALLEL DO directives begin with a legal Fortran comment character (either C or !). This is by design. In order to invoke the parallel procesing commands, you must turn on a specific switch in your makefile (this is -mp for SGI; check your compiler manual for other platforms). If you do not specify multiprocessor compilation, then the parallel processing directives will be considered as Fortran comments, and the associated DO-loops will be executed on one processor only. Also, since GEOS–CHEM uses the traditional "fixed-form" Fortran style, the !$OMP commands must begin at column 1. Otherwise a syntax error will result at compile time.
It should be noted that OpenMP commands are not the same as MPI (message passing interface). With OpenMP directives, you are able to split a job among several processors on the same machine. You are NOT able to split a job among several processors on different machines. Therefore, OpenMP is not suitable for Beowulf-style architectures. In 2003 we will look at the feasibility of porting GEOS–CHEM to an MPI environment. This may not be realized for quite some time, however.
For more information about OpenMP, consult the web site OpenMP.org. Also, there is a very good online tutorial at Boston University's Scientific Computing and Visualization department.