Debugging on SHARCNET

This semester, due to the master's work of Sid Bao, we have a new log visualization feature in Pilot. A lab handout describes how to use this. If you run into any issues using this feature, please post in CourseLink and/or contact the Instructor ASAP for help.

The mako cluster does not have an interactive debugger, but you can debug Pilot programs on the orca cluster using ddt , which is licensed software. Because these clusters' software environments are different, you will need to rebuild your Pilot application to run on orca (no source code changes are required). When logged into orca or mako, you will automatically get the right Pilot and MPE libraries for that cluster if you set $PILOTHOME to /work/wgardner/pilot and $MPEHOME to /work/wgardner/mpe in your makefile. (This works through symbolic links in /work/wgardner and /scratch/wgardner, the latter being cluster specific.)

  • 1. Login to orca.sharcnet.ca using "ssh -Y" so that ddt, an X-windows application, can open its window on your computer.
  • 2. Load the ddt module: "module load ddt"
  • 3. Check ddt (should print "/opt/sharcnet/ddt/ version /bin/ddt"): "which ddt"
  • 4. Insert this line in your makefile after CPPFLAGS to turn on debugging symbols and disable compiler optimization: "CFLAGS += -g -O0". The Pilot library was also compiled with these options.
  • 5. Assuming your program is called foo, do: "make clean" and "make foo"
  • 6. Start ddt in the background; it should open a window after several seconds: "ddt foo args &". The args could include -picheck=3, -pisvc=d, etc.
  • 7. A large window for ddt will open, with a small window titled "Run@orc..." in front of it. Check that the application and arguments are correct. Change the number of processes to whatever your want (same as -n argument of sqsub). Don't check any other boxes. Click Run.
  • 8. A "listening" window will open while your job is being started. Wait for it to disappear and for the GUI to fill with your source code. If you requested N processes, you will see red buttons labeled 0 to N-1 that you can use to redirect your attention to any of them. (PI_MAIN is process 0. Process 1 is the deadlock detector, if you enabled it. Other numbers are assigned in the same order as PI_CreateProcess statements.) Initially, all processes will be stopped at the PI_Configure() call in your application, and at the MPI_Init_thread() call in pilot.c.
  • 9. Go ahead and use the debugger in the usual way, setting breakpoints, inspecting variables, etc. You have a choice whether a breakpoint applies to a given single process or to all of them. The Input/Output tab at the bottom will show the output from stdout, including from Pilot.

    • It is possible to edit code in ddt. Just remember to type "make" in the orca shell, so your code is linked with orca's libraries and not mako's.

    For more help, see this DDT tutorial on SHARCNET.