ProbeVue Offers a Flexible Tracing and Debugging Tool
Tracing is an elementary step in system-activity analysis, where we collect the required data at a specific point in the software. This analysis is required in debugging any problem. With systems being loaded with new technologies/features day by day, system complexity increases. Not only the systems but also user applications are becoming more sophisticated so that they can utilize these technologies/features to the fullest. Because of this increase in complexity of systems and setup, tracing needs to be more precise and well planned. If not, users can be hung up with humongous amounts of data to analyze.
AIX has multiple tracing/debugging tools, including the kdb and dbx debug commands, and system trace and component trace. AIX also supports monitoring tools like tprof, curt, etc. Debuggers are the best way to trace any application as the user will have full control of the application and can fetch any information at the point of interest in the application. However, these come with disadvantages. For example, the application will halt while the user is collecting the data. Therefore, one can’t use this mode of tracing in production systems, because halting an application for even a second is unacceptable. Even in development environment, tracing becomes tedious since the application is going to halt every time the trace point is hit.
Tracing facilities, like system trace, component trace, etc., are very useful in collecting trace data. Users have the flexibility to enable or disable it at will. However, trace points and the data to be collected by these facilities are hard coded, i.e., users don’t have the freedom to modify these. Moreover, when you enable a trace point for even a specific thread, all the trace points across the whole code of the thread get enabled and start logging the data. Here, if users are interested in only a subset of functions, they will have a tough time filtering the data. Monitoring tools rely more or less on these system and component traces for their data collection, so there isn’t much difference in them.
Considering the nature of these mechanisms of collecting traces, AIX has a new tracing tool named ProbeVue to overcome these limitations. It’s a dynamic tracing facility that was introduced in AIX version 6.1. This tool is aimed at installing probes at any location or event in the system. To achieve this, ProbeVue supports probing of applications written in C/C++ and Java. Users can put trace points (probes) on exported system calls and base system calls as well. Because of its dynamic nature users can easily trace only the selected function and collect only the required data.
While tracing with ProbeVue, users need not pre-compile trace hooks with source code as compared to tracing facilities. Unless and until a user has initiated any tracing session with ProbeVue, the system is free from any ProbeVue trace points. It’s interesting to note that when trace points set by ProbeVue are hit, the data is collected and available for immediate consumption. Users need to specify the trace points and data to be collected with the help of new Vue language syntax. Data that can be collected are arguments to the function, register values, thread ID, process ID, etc. Users can place trace hooks at entry and exit of functions.
Consider a scenario in which one needs to track who is deleting a particular file from the system. Achieving this with existing tracing facilities is not a workable solution. But with ProbeVue, a user can safely put a trace point on system call “unlink” and fetch the first argument that contains the name of the file to delete. In probe action, the user can specify if the file being deleted is the one in question, then print the required data, like process ID, thread ID, process tree, etc. Here is a sample script for the same.
int unlink(char *fname);
__auto String fname;
fname = get_userstring(__arg1, -1);
printf("File %s is being unlinked by %u", fname, __pid);
* Displays the process tree of the process unlinking the file
This Vue script, puts a trace point at the entry of the system call, named unlink. Whenever this system call is called, the code written inside the brackets is executed. In this case, ProbeVue will fetch the file name that’s being deleted. Additionally, it prints the process tree, using a ptree API of Vue language.
Now, let’s look at a common problem in which users want to track who is sending a signal to a specific process. User can safely put a trace point on the kill system call and fetch its arguments, which are process ID (PID) and signal. If the PID passed to the kill system call is the one we’re looking for, we can print the required data. Here is the sample script:
/* Check if, signal is being sent to our process denoted with $1 */
if($1 == __arg1)
printf("Process : %s Sent Signal : %d to Pid : %d \n" , __pname,__arg2,__arg1);
printf("Below is further information on the process : \n");
printf(" Process id : %d\n",__pid);
printf(" Thread id : %d\n",__tid);
printf(" Parent Process id : %d\n",__ppid);
printf("------ Stack trace ------\n");
This script is designed to put a trace point at entry of the kill system call. Whenever the kill system call is called, the action block within the brackets is executed, and we check if, the PID passed to this kill system call is the same as what we’re interested in. If so, then the “if” statement is executed, and the user can get the required information. Here, we are printing the stack trace along with other information. Thereby, users can filter the data to what they’re interested in.