Wednesday, 28 June 2023
frinringAnd what about…
While talking about the build time improvements seen by avoiding the use of Qt module header includes Volker Krause wondered: in chat
regarding the compile time improvements, I have the suspicion that included moc files would help with incremental build times, possibly even quite noticeably (compiling the combined automoc file can be quite expensive), but no idea how that impacts clean builds
And while he was occupied with other things, this suspicion caught my interest and curiousity, so I found some slots to give it some closer look and also learn some more.
After all, people including myself had removed quite some explicit moc includes by the years, also in KDE projects, enjoying existing automoc magic for less manual code. Just that in the mean time, as soon noticed, Qt developers had stepped up efforts to add them for the Qt libraries were missing, surely for reasons.
Back to the basics…
Let’s take a simple example of two independent QObject sub-classes Foo and Bar, with own header and source files:
foo.h: class Foo : public QObject { Q_OBJECT /* ... */ };
bar.h: class Bar : public QObject { Q_OBJECT /* ... */};
foo.cpp: #include "foo.h" /* non-inline Foo method definitions */
bar.cpp: #include "bar.h" /* non-inline Bar method definitions */
CMake’s automoc will detect the respective Q_OBJECT macro usages and generate build system rules to have the moc tool create respective files moc_foo.cpp and moc_bar.cpp, which contains the code complementing the macro (e.g. for the class meta object).
CMake then, if no source files include those generated moc files, will have added rules to generate for each library or executable target a central file mocs_compilation.cpp which includes those:
// This file is autogenerated. Changes will be overwritten.
#include "<SOURCE_DIR_CHECKSUM>/moc_foo.cpp"
#include "<SOURCE_DIR_CHECKSUM>/moc_bar.cpp"This results in a single compilation unit with all the moc code. It is faster to build compared to compiling all moc files in separate ones. Note the “all” here, as all moc code only needs to be build together in full project (re)builds.
Incremental build, wants to handle minimal size of sources
When working on a codebase, one usually does incremental builds, so only rebuilding those artifacts that depend on sources changed. That gives quick edit-build-test cycles, helping to keep concentration stable (when no office-chair sword duel tournaments are on-going anyway).
So for the example above when the header foo.h is edited, in an incremental build…
- the file foo.cpp is recompiled as it includes this header…
- next moc_foo.cpp to be regenerated from the header and then…
- mocs_compilation.cpp to be recompiled, given it includes moc_foo.cpp.
Just, as mocs_compilation.cpp does not only include moc_foo.cpp, but also moc_bar.cpp, this means also the code from moc_bar.cpp is recompiled here, even if does not depend on foo.h.
So the optimization of having a single compilation unit for all moc files for headers, done for full builds, results in unneeded extra work for incremental builds. Which gets worse with any additional header that needs a moc file, which then also is included in mocs_compilation.cpp. And that is the problem Volker talked about.
Impact of mocs_compilation.cpp builds
On the author’s system (i5-2520M CPU @ 2.5 GHz, with SSD) some measurements were done by calling touch on a mocs_compilation.cpp file (touch foo_autogen/mocs_compilation.cpp), then asking the build system to update the respective object file and measuring that with the tool time (time make foo_autogen/mocs_compilation.cpp.o).
To have some reference, first a single moc file of a most simple QObject subclass was looked at, where times averaged around 1.6 s. Then random mocs_compilation.cpp found in the local build dirs of random projects were checked, with times measured in the range of 5 s to 14 s.
Multiple seconds spent on mocs_compilation.cpp, again and again, those can make a difference in the experience with incremental builds, where the other updates might take even less time.
Impact of moc include on single source file builds
Trying to measure the cost which including a moc file adds to (re)compiling a single source file, again the tool time was used, with the compiler command as taken from the build system to generate an object file.
A few rounds of measurement only delivered average differences that were one or two magnitudes smaller than the variance seen in the times taken, so the cost considered unnoticeable. A guess is that the compiler for the moc generated code added can reuse all the work already done for the other code in the including source file, and the moc generated code itself not that complicated relatively.
This is in comparison to the noticeable time it needs to build mocs_compilation.cpp, as described above.
Impact of moc includes on full builds, by examples
An answer to “no idea how that impacts clean builds” might be hard to derive in theory. The effort it takes to build the moc generated code separately in mocs_compilation.cpp versus the sum of the additional efforts it takes to build each moc generated code as part of source files depends on the circumstances of the sources involved. The measurements done before for mocs_compilation.cpp and single source files builds though hint to overall build time reduction in real-world situations.
For some real-world numbers, a set of patches for a few KDE repos have been done (easy with the scripts available, see below). Then some scenario of someone doing a fresh build of such repo using the meta-build tool kdesrc-build was run a few times on an otherwise idle developer system (same i5-2520M CPU @ 2.5 GHz, with SSD), both for the current codebase and then with all possible moc includes added.
Using the make tool, configured to use 4 parallel jobs, with the build dir always completely removed before, and kdesrc-build invoked with the –build-only option, so skipping repo updates, the timing was measured using the time tool as before. Which reports by “real” the wall clock timing, while “user” reports the sum of times of all threads taken in non-kernel processor usage. The time spent by related kernel processing (“sys”) was ignored due to being very small in comparison.
The numbers taken in all cases showed that there clean builds got faster with moc includes, with build times partially reduced by more than 10 %:
| Before | “moc includes” | Reduction | Average of | Variance | ||
| LibKDEGames (MR) | real | 1m 02,18s | 0 min 58,46 s | 6 % | 5 runs | 2 s |
| user | 3m 06,37s | 2 min 48,13 s | 10 % | 3 s | ||
| KXmlGui (MR) | real | 2 min 26,62 s | 2 min 09,09 s | 12 % | 3 runs | – |
| user | 7 min 34,42 s | 6 min 35,07 s | 13 % | – | ||
| Kirigami (MR) | real | 1 min 32,83 s | 1 min 29,79 s | 3 % | 3 runs | – |
| user | 4 min 25,67 s | 4 min 19,94 s | 2 % | – | ||
| NetworkmanagerQt (MR) | real | 11 min 48,10 s | 11 min 18,57 s | 4 % | 1 run | – |
| user | 40 min 39,78 s | 39 min 05,28 s | 4 % | – | ||
| KCalendarCore (MR) | real | 3 min 09,91 s | 2 min 42,83 s | 14 % | 3 runs | – |
| user | 10 min 17,57 s | 8 min 54,90 s | 13 % | – |
Further, less controlled own time measurements for other codebases support this impression, as well as reports from others (“total build time dropped by around 10%.”, Qt Interest mailing list in 2019). With that for now it would be assumed that times needed for clean build are not a reason against moc includes, rather the opposite.
And there are more reasons, read on.
Reducing need for headers to include other headers
moc generated code needs to have the full declaration of types used as values in signals or slots method arguments. Same for types used as values or references for Q_PROPERTY class properties, in Qt6 also for types used with pointers:
class Bar; // forward declaration, not enough for moc generated code here
class Foo : public QObject {
Q_OBJECT
Q_PROPERTY(Bar* bar READ barPointer) // Qt6: full Bar declaration needed
Q_PROPERTY(Bar& bar READ barRef) // full Bar declaration needed
Q_PROPERTY(Bar bar READ barValue) // full Bar declaration needed
Q_SIGNALS:
void fooed(Bar bar); // full Bar declaration needed
public Q_SLOT:
void foo(Bar bar); // full Bar declaration needed
// [...]
};So if the moc file for class Foo is compiled separately and thus only sees the given declarations as above, if will fail to build.
This can be solved by replacing the forward declaration of class Bar with the full declaration, e.g. by including a header where Bar is declared, which itself again might need more declarations. But this is paid by everything else which needs the full class Foo declaration now also getting those other declarations, even if not useful.
Solving it instead by including the moc file in a source file with definitions of class Foo methods, with full class Bar declaration available there, as usually already needed for those methods, allows to keep the forward declaration:
#include "foo.h"
#include "bar.h" // needed for class Foo methods' definitions
// [definitions of class Foo methods]
#include "moc_foo.cpp" // moc generated code sourcedWhich keeps both full and incremental project builds faster.
In KDE projects while making them Qt6-ready a set of commits with messages like “Use includes instead of forward decl where needed” were made, due to the new requirements by moc generated code with pointer types and properties. These would not have been needed with moc includes.
Enabling clang to warn about unused private fields
The clang compiler is capable to check and warn about unused private class members if it can see all class methods in the same compilation unit (GCC so far needs to catch up):
class Foo : public QObject {
Q_OBJECT
/* ... */
private:
bool m_unusedFlag;
};The above declaration will see a warning if the moc file is included with the source file having the definition of all (normal) non-inline methods:
/.../foo.h:17:10: warning: private field 'm_unusedFlag' is not used [-Wunused-private-field] boolm_unusedFlag; ^
But not if the moc file is compiled separately, as the compiler has to assume the other methods might use the member.
Better binary code, due to more in the compilation unit
A moc include into a source file provides the compiler with more material in the same compilation unit, which is said to be usable for some optimizations:
- “but also it produces slightly better code overall.” (Qt Interest mailing list, Dec. 2019)
- “Plus the benefit of more inlining, as there’s more the compiler can see.” (Qt Development mailing list, May 2020)
Indeed when building libraries in Release mode, so with some optimization flags enabled, it can be observed that size shrank by some thousandths for some. So at least size was optimized. For others though it grew a tiny bit, e.g. in the .text section with the code. It is assumed this is caused by the code duplications due to inlining. So there runtime is optimized at the cost of size, and one would have to trust the compiler for a sane trade-off, as done with all the other, normal code.
For another example, one of the commits to Qt’s own modules establishing moc includes for them reports in the commit message for the QtWidgets module:
A very simple way to save ~3KiB in .text[edit] size and 440b in data size on GCC 5.3 Linux AMD64 release builds.
So far it sounds like it is all advantages, so what about the disadvantages?
More manual code to maintain with explicit moc include statements
To have explicit include statements for each moc file covering a header (e.g. moc_foo.cpp for foo.h) means more code to manually maintain. Which is less comfortable.
Though the same is already the case for moc files covering source files (e.g. foo.moc for foo.cpp), those have to be included, given the class declarations they need are in that very source file. So doing the same also for the other type would feel not that strange.
The other manual effort needed is to ensure that any moc include is also done. At least with CMake’s automoc things will just silently work, any moc file not explicitly included is automatically included by the target’s mocs_compilation.cpp file. That one is currently always generated, built and linked to the target (TODO: file wish to CMake for a flag to have no mocs_compilation.cpp file).
One approach to enforce moc includes might be to add respective scripts as commit hooks, see. e.g. check-includemocs-hook.sh from KDAB’s KDToolBox.
No longer needed moc includes are also not critical with CMake’s automoc, an empty file will be generated and a warning added to the build log. So the developer can clean-up later when there is time.
So the cost is one include statement per moc-covered header and its occasional maintenance.
Automated moc file include statements addition, variant 6
There exist already a few scripts to scan sources and amend include statements for moc files where found missing, like:
- includemocs from the KDE Development Scripts repo (as old as 23 (sic!) years, see introducing commit)
- includemocs.pl & includemocs6.sh from Qt’s Qt Base repo util subdir
- Includemocs from KDAB’s KDToolBox repo
- fixMocInclude.sh from Remy van Elst’s blog post Add moc includes to speed up Qt compilation
Initially I was not aware of all. The ones tested (KDE’s, KDAB’s & Remy van Elst’s) missed to cover matching header files with the basename suffixed by “_p” (e.g. foo_p.h) to source files without that suffix (e.g. foo.cpp). So there is now a (working for what used for) draft of yet another script, addmocincludes. Oh dear 
Suspicion substantiated: better use moc includes
As shown above, it looks that the use of explicit includes also for header moc files improves things for multiple stakeholders:
- developers: gain from faster full & incremental builds, more sanity check
- users: gain from runtime improvements
- CI: gains from faster full builds
- packagers: gain from faster full builds
All paid by the cost of one explicit include statement for each moc-covered header and its occasional maintenance. And in some cases a slightly bigger binary size.
Seems a good deal, no? So…
- pick of one the scripts above and have it add more explicit moc includes
- check for some now possible forward declarations
- look out for any newly discovered unused private members
- PROFIT!!! (enjoy the things gained long term by this one-time investment)
Update (Aug 14th):
To have the build system work along these ideas, two issues have now been filed with CMake’s issue tracker:
@vkohli17:matrix.org
GSoC
alex1701c
