Bruce Family

User Tools

Site Tools

Trace: • postgres_stack_traces
software:postgres_stack_traces

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
software:postgres_stack_traces [2025/10/23 01:05] – [Postgres Stack Traces] timbsoftware:postgres_stack_traces [2025/10/23 01:10] (current) – timb
Line 7: Line 7:
 Attached are annotated examples of: Attached are annotated examples of:
  
- * how to capture stack traces using the helper script+* how to capture stack traces using the helper script
  
- * a few tips on finding useful facts in the stack traces+* a few tips on finding useful facts in the stack traces
  
-== How to Capture ==+==== How to Capture ====
  
 <code> <code>
Line 20: Line 20:
 </code> </code>
  
-== How to Analyze ==+==== How to Analyze ====
  
- * For reference, the output files are usually ~100-1000 KB.+* For reference, the output files are usually ~100-1000 KB.
  
 <code> <code>
Line 59: Line 59:
  * There are currently 2 helper scripts to simplify and filter the stacks.  * There are currently 2 helper scripts to simplify and filter the stacks.
  
-   * ``stack_trace_filter.simplify_frames.pl``+ * ``stack_trace_filter.simplify_frames.pl``
  
  * Strips out all arguments from the function.  Does not remove other lines.  Intended as input for other filter script.  * Strips out all arguments from the function.  Does not remove other lines.  Intended as input for other filter script.
Line 65: Line 65:
  * Will consolidate at some point in the future.  * Will consolidate at some point in the future.
  
-   * ``stack_trace_filter.regexp_match_on_whole_stack.pl``+ * ``stack_trace_filter.regexp_match_on_whole_stack.pl``
  
  * Show whole stack if any frame matches the given perl regexp.  Outputs 1 line per frame.  * Show whole stack if any frame matches the given perl regexp.  Outputs 1 line per frame.
Line 72: Line 72:
  
 <code> <code>
- 
 [postgres@db04.prd ~]$ ls -l ~/bin/stack_trace_filter.* [postgres@db04.prd ~]$ ls -l ~/bin/stack_trace_filter.*
 -rwxr-xr-x. 1 postgres postgres 347 Jan 10 18:24 /var/lib/pgsql/bin/stack_trace_filter.regexp_match_on_whole_stack.pl -rwxr-xr-x. 1 postgres postgres 347 Jan 10 18:24 /var/lib/pgsql/bin/stack_trace_filter.regexp_match_on_whole_stack.pl
 -rwxr-xr-x. 1 postgres postgres 117 Jan 10 22:27 /var/lib/pgsql/bin/stack_trace_filter.simplify_frames.pl -rwxr-xr-x. 1 postgres postgres 117 Jan 10 22:27 /var/lib/pgsql/bin/stack_trace_filter.simplify_frames.pl
- 
 </code> </code>
  
  
- * Examples:+* Examples:
  
- * Review each stack individually, but without the noise of the full stack trace output.+* Review each stack individually, but without the noise of the full stack trace output.
  
 <code> <code>
- 
 [postgres@db04.prd ~]$ cat /db/scratch/stack_traces.20130116_165754.out | ~/bin/stack_trace_filter.simplify_frames.pl | ~/bin/stack_trace_filter.regexp_match_on_whole_stack.pl '' | less [postgres@db04.prd ~]$ cat /db/scratch/stack_traces.20130116_165754.out | ~/bin/stack_trace_filter.simplify_frames.pl | ~/bin/stack_trace_filter.regexp_match_on_whole_stack.pl '' | less
 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------
Line 190: Line 187:
 </code> </code>
  
- * Aggregate frames from all stacks.  Sort by frame number.+* Aggregate frames from all stacks.  Sort by frame number.
  
- * This is helpful for seeing what are the most common frames among all the stacks.  Very helpful for performance profiling.+* This is helpful for seeing what are the most common frames among all the stacks.  Very helpful for performance profiling.
  
- * Particularly helpful for:+   * Particularly helpful for:
  
-   * Finding where a single process is spending most of its time, by capturing its stack multiple times during a single query or session, and aggregating the stack traces.  Frequently occurring frames or functions indicate common places where time is spent.+       * Finding where a single process is spending most of its time, by capturing its stack multiple times during a single query or session, and aggregating the stack traces.  Frequently occurring frames or functions indicate common places where time is spent.
  
-   * Note that a function may appear in different frame numbers, depending on when the process was interrupted.+       * Note that a function may appear in different frame numbers, depending on when the process was interrupted.
  
-   * The same trick can be used on many processes if they are doing the same thing (e.g. all testtaker sessions behave similarly).+       * The same trick can be used on many processes if they are doing the same thing (e.g. all testtaker sessions behave similarly).
  
-   * Interpreting system-call traces (strace).  Strace shows how often and how long postgres is making system calls, but the next question is why are those calls prevalent.  To answer that question, capture a series of stack traces and look for ones where one of the top few frames is a system call (read, recv, semop, etc.).+  * Interpreting system-call traces (strace).  Strace shows how often and how long postgres is making system calls, but the next question is why are those calls prevalent.  To answer that question, capture a series of stack traces and look for ones where one of the top few frames is a system call (read, recv, semop, etc.).
  
 <code> <code>
 +.
 [postgres@db04.prd ~]$ cat /db/scratch/stack_traces.20130116_165754.out | ~/bin/stack_trace_filter.simplify_frames.pl | ~/bin/stack_trace_filter.regexp_match_on_whole_stack.pl '' | sort -k2 -n | uniq -c | less [postgres@db04.prd ~]$ cat /db/scratch/stack_traces.20130116_165754.out | ~/bin/stack_trace_filter.simplify_frames.pl | ~/bin/stack_trace_filter.regexp_match_on_whole_stack.pl '' | sort -k2 -n | uniq -c | less
 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------
software/postgres_stack_traces.1761181531.txt.gz · Last modified: by timb