Differences

This shows you the differences between two versions of the page.

--- gsoc:2024-gsoc-perf [2024/01/12 19:03]
till created
+++ gsoc:2024-gsoc-perf [2024/02/08 05:08] (current)
namhyung [Data type profiling]
@@ Line 15: / Line 15: @@
 ===== Perf community =====
-[[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS|Maintainers]]: Peter Zijlstra <peterz at infradead.org>, Ingo Molnar <mingo at redhat.com>, Arnaldo Carvalho de Melo <acme at kernel.org>
+[[https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/MAINTAINERS|Maintainers]]: Peter Zijlstra <peterz at infradead.org>, Ingo Molnar <mingo at redhat.com>, Arnaldo Carvalho de Melo <acme at kernel.org>, Namhyung Kim <namhyung at kernel.org>
 Mailing list: [[https://www.spinics.net/lists/linux-perf-users/|linux-perf-users at vger.kernel.org]]
@@ Line 25: / Line 25: @@
 Wiki: [[https://perf.wiki.kernel.org/index.php/Main_Page|https://perf.wiki.kernel.org/]]
-Mentor contacts: [[https://sites.google.com/site/rogersemail/home|Ian Rogers]] <irogers+gsoc22 at google dot com>, Namhyung Kim <namhyung at kernel.org>
+Mentor contacts: [[https://sites.google.com/site/rogersemail/home|Ian Rogers]] <irogers+gsoc24 at google dot com>, Namhyung Kim <namhyung at kernel.org>, Arnaldo Carvalho de Melo <acme at kernel.org>
 ===== Qualities of a good proposal =====
   * Contributor has engaged with the community by, for example, writing a patch, contributing to the wiki, or reporting a bug.
-  * The time plan for the project is clear and mentions how other commitments the contributor has will be managed alongside the project.
+  * The time plan for the project is clear and mentions how other commitments the contributor has will be managed alongside the project. Commitments likely vary over the some. As per GSoC projects are large (350h), medium (175h) or small (90h) so show how you will use the time.
   * There is sufficient detail in the proposal that it is clear the contributor and mentors will be able to get the project done.
@@ Line 37: / Line 37: @@
 ==== Bring your open proposal ====
-If you have your own ideas for how tracing and profiling can be improved in the Linux kernel and perf tool then these are welcomed. Some areas that have been brought up in the last year are [[https://lore.kernel.org/linux-perf-users/87o85ftc3p.fsf@smart-cactus.org/|better support for more programming languages]] and [[https://lore.kernel.org/all/20211129231830.1117781-1-namhyung@kernel.org/|new profiling commands like function latency measuring]].
+  * complexity: intermediate or hard
+  * duration: small, medium or large
+  * requirements: machine to work and test on, typically a bare metal (ie not cloud) Linux machine. C programming, possibly other languages if interested in things like Rust integration.
-==== User Interface ====
+If you have your own ideas for how tracing and profiling can be improved in the Linux kernel and perf tool then these are welcomed. Some areas that have been brought up in the last year are [[https://lore.kernel.org/linux-perf-users/87o85ftc3p.fsf@smart-cactus.org/|better support for more programming languages]] and [[https://lore.kernel.org/all/20211129231830.1117781-1-namhyung@kernel.org/|new profiling commands like function latency measuring]]. The perf tool is full of metrics like flops and memory bandwidth, so adding a [[https://www.inesc-id.pt/ficheiros/publicacoes/9068.pdf|roofline model]] to determine the bottlenecks of an application would be a possibility.
-=== Improved perf trace ===
+==== New Performance Monitoring Unit (PMU) kernel drivers ====
-''perf trace'' provides an strace like ability using perf APIs. By avoiding ptrace APIs, ''perf trace'' can even trace itself. The filename for an open system call is fairly boring encoded as a pointer. To get the actual filename the program is sending to the kernel requires information not captured in a perf event, and so BPF and a list of argument types is maintained. This project will look to update the BPF code in ''perf trace'' so that it is shipped as a skeleton built into the actual perf application, rather than as a file on the command line. Further, the project will look to automate the task of decorating syscall arguments using the BPF Type Format (BTF).
+  * complexity: intermediate or hard
+  * duration: small, medium or large
+  * requirements: machine to work and test on, typically a bare metal (ie not cloud) Linux machine, test hardware for the PMU you are working on. C programming.
-=== Interactive perf report ===
+Have a computer you really love but can only query the core CPU's PMU? Are there data sheets describing performance monitoring counters that could be exposed through the perf event API? Why not work to add a PMU driver to the Linux kernel and expose those performance counters or even more advanced features like sampling. Drivers can be added for accelerators, GPUs, data buses, caches, etc. For example, the Raspberry Pi 5 has performance counters only for its core CPUs and not for things like its memory bus.
-Currently the ''perf report'' command will process an entire perf.data file before providing a visualization. This can be slow for large perf.data files. In contrast, the ''perf top'' command will gather data in the background while providing a visualization. This project will look to break apart ''perf report'' so that processing is performed on a background thread with the visualization periodically refreshing.
+==== Improved Python integration ====
-=== Adding or improving graphical toolkit support ===
+  * complexity: intermediate or hard
+  * duration: small, medium or large
+  * requirements: machine to work and test on, typically a bare metal (ie not cloud) Linux machine. C and/or Python programming.
-Perf has different user interfaces:
+A lot of what makes the perf tool useful is user interface, however, writing user interfaces in C is tedious and error prone. Python support is long established within the perf tool but it could use some TLC. Some examples of work that needs doing are:
-  * stdio - the most basic UI that can use a pager like ''less'',
+  * Making the python module depend only on libperf. libperf is a library of perf's core components. Currently the python module uses bits and pieces of code from the perf code base but with bits stubbed out. Making the python module just depend on libperf and cleaning up libperf would solve this.
-  * tui - based on [[http://www.jedsoft.org/slang/|slang]],
+  * A perf data file module that is standalone. Currently using the perf module to read a perf data file requires the C code. This is a distribution hassle and reading a file doesn't need the rest of perf. [[https://android.googlesource.com/platform/system/extras/+/refs/heads/main/simpleperf/scripts|Other projects]] have written perf data file readers in python, but they suffer as the perf.data file format evolves. The perf tool's testing can ensure the python and C implementations are in sync. Ideally such a module can be installed with tools like ''pip'' so it is easy to depend upon.
-  * gtk - uses the gtk2 user interface. This option isn't enabled by default at build time.
+  * Improved packaging and testing. Most python scripts in perf are under the perf script command, but some like ''flamegraph'' and ''gecko'' deserve to be top-level. Improving how this is organized should help perf users. The current command line argument processing for ''perf script'' is also messy as arguments may be for record, report or both.
+  * Improved testing. As with perf's C code, good code coverage and testing will ensure the python integration is working well. Current testing is limited to just things like did the python module load.
-This project will look to improve the user interface experience with perf, for example, by using a new user interface library like [[https://www.textualize.io/|Textualize]]. The addition of other user interfaces would be an option for this project.
+With better python integration it is hoped that tools like perf report can also have a python equivalent. This would allow UI toolkits like [[https://www.textualize.io/|Textualize]] to be used and improve the user experience.
-==== Perf data converter ====
+A better python experience can also look to improve the gecko profiling experience or add other data converters like for pprof and Chrome's [[https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview|trace event format]].
-The ''perf data'' command allows a perf.data file to be converted to other file formats, however, currently only CTF is supported. This project will look to add support for other file formats such as Firefox's [[https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md|Gecko profile format]] or Chrome's [[https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview|trace event format]]. By supporting conversion to these file formats visualization tools will be able to show a recorded perf.data file.
 ==== Scalability and speed ====
-Large parts of the perf command are single threaded. As CPU core counts increase, this means that programming events can be slow and the command in general can be slower that it needs to be. This project will look to provide parallel approaches to processing perf data structures such as ''evlist'' so that more concurrency happens within the perf command. This work will build upon that done by Riccardo Mancini [[https://lore.kernel.org/linux-perf-users/3c4f8dd64d07373d876990ceb16e469b4029363f.camel@gmail.com/|in GSoC 2021]].
+  * complexity: intermediate or hard
+  * duration: small, medium or large
+  * requirements: machine to work and test on, typically a bare metal (ie not cloud) Linux machine. C programming, multi-threading/pthread library.
+The perf tool is largely single threaded even though sometimes it needs to do something on every CPU in the system. This is embarrassingly parallel but the tool isn't exploiting it. Work was done to create a [[https://lore.kernel.org/lkml/3c4f8dd64d07373d876990ceb16e469b4029363f.camel@gmail.com/|work pool mechanism]] but not merged due to latent bugs in memory management. [[https://perf.wiki.kernel.org/index.php/Reference_Count_Checking|Address sanitizer and reference count checking]] have solved this problem but we still need to integrate the work pool code.
+Another improvement is that currently the ''perf report'' command will process an entire perf.data file before providing a visualization. This can be slow for large perf.data files. In contrast, the ''perf top'' command will gather data in the background while providing a visualization. Breaking apart the ''perf report'' command so that processing is performed on a background thread with the visualization periodically refreshing in the foreground will mean that at least during the slow load the user can do something.
+One more thing can do is to reduce the number of file descriptors in ''perf record'' with ''--threads'' option.  Currently it needs a couple of pipes to communicate between the worker threads.  I think it can be greatly reduced by using eventfd(2) instead of having pipes for each thread.
+==== Data type profiling ====
+  * complexity: intermediate or hard
+  * duration: medium or large
+  * requirements: physical machine to work and test on (Intel recommended). C programming, understanding DWARF format is a plus.
+Data type profiling is a new technique to show memory access profiles with type information.  See [[https://lwn.net/Articles/955709/ | LWN article]] for more detail.  It's still in the early stage and has a lot of room for improvement.  For example, it needs to support C++ and other languages, better integration to other perf commands like ''annotate'' and ''c2c'', performance optimization, other architecture support and so on.  It'd be ok if you're not familiar with ELF or DWARF format.
+==== perf trace and BTF ====
+  * complexity: intermediate or hard
+  * duration: medium or large
+  * requirements: machine to work and test on. C programming, BPF
+''perf trace'' is similar to ''strace'' but much performant since it doesn't use ptrace.  So it needs to capture and understand the format of syscall arguments as ''strace'' does.  Right now, it has to build a list of format to pretty-print the syscall args.  But we find it limited and manual work.  Instead, it could use BTF (BPF type format) which has all the type information and is available in the (most) kernel.

Wiki

User Tools

Site Tools

Differences

Page Tools