CS 595 - Advanced Scientific Computing

Fall 2010

The Boss is: OUT

Homework 6

1. Read chapters in Numerical Linear Algebra. If you are having trouble understanding the ideas (and believe me, this is tough stuff) I'd be happy to discuss ideas with you or point you to other references.
• Lecture 40: Preconditioning
2. Complete some experiments on solvers and preconditioners. The instructions below can also be found here.
   • Solver Report - This file contains a list of experiments you need to conduct, along with important data that you need to record.
   • Conducting these experiments is very straightforward as long as you are careful with what you type. The first thing you need to do is produce a build of PETSc in optimized mode, which is a version of PETSc that skips some safeguards in favor of running more quickly. Some of the experiments you're going to conduct will ask you to look at the difference between these two builds.
     
     To build in optimized mode, you need to go back to your PETSC_DIR and use a different PETSC_ARCH than your original build:
     cd $PETSC_DIR
     export PETSC_ARCH=arch-cs595-o
     ./config/configure.py --with-cc=gcc --with-fc=g77 --download-mpich=1 --download-f-blas-lapack=1 --with-debugging=0
     make all
     Now that you have a new PETSC_ARCH available to you, make sure that you are using the correct one for all your experiments. Now go to the ksp source library
     cd $PETSC_DIR/src/ksp/ksp/examples/tutorials
     and build ex2 using the debugging arch
     export PETSC_ARCH=arch-cs595
     make ex2
     If instead we were interested in running with the optimized arch, we would need to build with the commands
     export PETSC_ARCH=arch-cs595-o
     make ex2
     To run this, you have to make sure that you use the correct mpiexec (the one associated with your current arch), so I would recommend doing this by
     $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n <np> ./ex2 -m 300 -n 300 -log_summary <log_file>
     where <np> is the number of processors, and <log_file> is where PETSc writes the log data. You can call <log_file> whatever you want (I recommend petsc.log) and you'll need to look through that to fill in the table for the homework.
     
     In addition to those options that will need to appear on all your runs, you'll also need some other options to fill out the homework. Those options include
     1. -ksp_type gmres -pc_type none -ksp_max_it 1000
     2. -ksp_type cg -pc_type none -ksp_max_it 1000
     3. -ksp_type gmres -pc_type bjacobi -sub_pc_type ilu
     4. -ksp_type gmres -pc_type bjacobi -sub_pc_type lu
     5. -ksp_type cg -pc_type bjacobi -sub_pc_type icc
     6. -ksp_type gmres -pc_type asm -sub_pc_type lu
     7. -ksp_type gmres -pc_type asm -sub_pc_type lu -pc_asm_overlap 2
     8. -ksp_type cg -pc_type sor -pc_sor_local_symmetric
     Because the problem we are studying is linear (an Ax=b problem) the Dominating Operation required in the homework is going to be a matrix operation. As an example, consider the following output from a log_summary when ex2 was run with only one processor
     ./ex2 -m 300 -n 300 -ksp_type gmres -pc_type bjacobi -sub_pc_type ilu -log_summary |grep Mat
     
     

     MatMult	580	1.0 3.9221e+00 1.0 4.68e+08 1.0 0.0e+00 0.0e+00 0.0e+00 21 11 0 0 0 21 11 0 0 0 119
     MatSolve	580	1.0 4.0558e+00 1.0 4.68e+08 1.0 0.0e+00 0.0e+00 0.0e+00 21 11 0 0 0 22 11 0 0 0 115
     MatLUFactorNum	1	1.0 4.0746e-02 1.0 8.06e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 20
     MatILUFactorSym	1	1.0 4.9145e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00 0 0 0 0 0 0 0 0 0 0 0
     MatGetRowIJ	1	1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
     MatGetOrdering	1	1.0 3.2486e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 0 0 0 0 0 0 0
     MatAssemblyBegin	1	1.0 1.9073e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
     MatAssemblyEnd	1	1.0 1.8147e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0

     From this we can deduce that the most expensive operations are MatMult and MatSolve since the time spent in those is much greater than the other operations. In reality there are other things that need to be considered, but for the purposes of this assignment, that level of analysis is sufficient.
     
     Now that we've gotten through the linear example, we also need to run a nonlinear example. This can be found in the SNES directory:
     cd $PETSC_DIR/snes/examples/tutorials
     export PETSC_ARCH=arch-cs595
     make ex19
     This example allows you to run with multigrid on, so to get an idea of what's going on, try running with the following options
     ./ex19 -dmmg_nlevels 7 -ksp_monitor -snes_monitor -snes_view
     and checking out the -snes_view. It's not entirely straightforward, especially if you're not entirely comfortable with multilevel solvers to start with, but see what you can do.
     
     When you're feeling good, run with multiple processors using
     $PETSC_DIR/$PETSC_ARCH/bin/mpiexec -n <np> ./ex19 -dmmg_nlevels 7 -log_summary <log_file>
     Within the <log_file> this time, we'll be looking for all the matrix operations the same as for the linear example, but also
     SNESLineSearch
     SNESFunctionEval
     SNESJacobianEval
     as these are the key components (along with the KSP) to a nonlinear solve.