Rbuild sucks, let's rew... improve it
From ReactOS
Rbuild, our build system, has served us well so far, but its usefulness has since been vastly overshadowed by its awfulness. It's time to take action.
Phase 1: Denial
"But rbuild is prefect!"
Enjoy recompiling the whole tree for the thousandth time today.
Phase 2: Whining
Wherein developers air their grievances about rbuild, propose wonderful features nobody will implement, and generally moan and cry and say "shucks" a lot.
Proper support for Microsoft compilers
Official builds of Mozilla applications are compiled with Microsoft tools, and they use an autoconf-based build system. We should be ashamed. At present, all rbuild can do is create a whole lot of project files (one for each module), and a big solution file that links all of them in a monster that falls just short of crashing your Visual Studio, and produces an Intellisense database several hundred megabytes in size. This is so sad it isn't even funny.
Rbuild should have a backend that creates the usual huge makefile.auto, but one that invokes Microsoft compilers and linkers instead of the GNU toolchain. Missing, buggy or otherwise troublesome tools in Microsoft products shall be replaced with our own versions. One day this might even enable us to use other compilers with only moderate amounts of pain!
Separate build environments
Sometimes, it's convenient to write modules that by default only depend from standard SDK headers, and get access to internal ReactOS headers or library headers only when they ask for them. In fact, the vast majority of modules should be written like this - DLLs that only depend on the Win32 SDK, drivers that only need the DDK, WDK or WDM SDK, and so on. This is made hard by the unhealthy header soup we feed to even the simplest of our code
Achieving this would ensure clean code that can be independently recompiled with minimal hassle outside of the ReactOS project. It would also allow us to replace our embarrassingly buggy and incomplete versions of the public SDKs (that don't even support other compilers than GCC) with the Microsoft versions (which might be in violation of some stupid draconian EULA, but hey. Technology. What's not to love). In general, it would allow people to actually work on our public SDKs like they were subprojects of their own, as opposed to ugly roadblocks we all get to stumble into, and try to forget after the obligatory two-line patch that unbreaks the build. These brave individuals should be able to change a line in the rbuild configuration file to compile a build using public SDKs wherever possible and compare the output with a build compiled the good ol' way
Support for Microsoft tools and environments should not mean "Yay, we get to do less work by offloading the boring parts to Microsoft". We write an operating system, and writing an SDK for it is our responsibility. This change would make working on our SDKs possible
The elusive ReactOS Development Kit
We should provide an SDK for ReactOS because it's the right thing to do. Failing the moral argument, we should at least pretend we care, and do the work anyway, "as if". Maybe the end result will not be an usable SDK, but at least we will have abstracted the SDK headers well enough that an external SDK can be used.
Fixing the header hell, in general
The answer to fixing the headers is obvious: find each thing that would be multiply defined as a group, make a single header that defines that thing, and include it everywhere it would be duplicated. Things like NtCurrentTeb, NTSTATUS codes, exception definitions, and shared definitions like IN, INLINE, ULONG etc should be handled this way. If we survived the ensuing revert war, we'd be better off.
No external dependencies
We need to provide all libraries needed to compile our modules. We cannot depend on internal libraries shipped with the compiler. Rbuild should have an option to disable any and all references to libraries and headers external to our tree, to ensure we know exactly what kind of shit gets put in our binaries
Consistency checks for internal dependencies
Ntdll.dll must have no external dependencies. Kernel32.dll should only ever import from ntdll.dll. Native executables must not link Win32 ones. Kernel mode executables must not link user mode ones, and vice versa. If you think this sounds obvious, you haven't been in this project long enough to have seen ntdll.dll importing functions from msvcrt.dll. Rbuild doesn't prevent this. Its fancy XML syntax and object-oriented code are only for show: the thing just pastes text fragments together and it's about as smart as roadkill. In fact, its default rules sometimes - oops! - forget to omit some -nosomething or -fnoanother flag, that could cause the linker to pull in the wrong library (possibly from its internal set, see "No external dependencies" above) and introduce the wrong dependency.
Another cool thing that rbuild really should do but doesn't would be checking for base address conflicts. Wouldn't it be cool to identify a problem that can make ReactOS unbootable during that boring, endless phase where rbuild gurumeditates your disk into fairy dust? All that overhard work would not be in vain.
Proper support for dependencies
Tracking dependencies is expensive. Tracking dependencies melts your hard disk. Tracking dependencies kicks your dog. Tracking dependencies prevents your whole tree from rebuilding every single damn time you do a SVN update, though. And it makes parallel building possible. We have a 8-core build server that sits three-quarters-unused most of the time, and seven-eighths sometimes. Surely there has to be a way we can make dependency tracking more useful and less traumatic. Microsoft's internal "build" keeps a central database of dependencies, why can't we too?
While we're at it, object files should have a dependency on compiler flags, and executable files on linker flags, and so on. Which is, naturally, impossible to do as long as some flags come from logic external to the rbuild files...
More abstraction
Ban string pasting
String pasting is evil. It's inefficient, it gives a false illusion of flexibility, it allows mistakes and misuse. The current rbuild format is an egregious abuse of XML. A loose schema, macro expansion in attributes, attributes used instead of elements, conditional compilation based on data and logic external to the XML (<if property="ARCH" value="i386">? holy overengineering Batman, how about a simple arch="i386" that you can apply to any element, you know, something you can actually process with XSLT?), rules hardcoded in the build driver's source... a complete and utter mess. Some of it is salvageable, yes, some issues are just "style", but there's a lot of work to do in this area, and it's important.
More module types
Something for type libraries, like the recent (April-May 2008) MSI fiasco showed. A module type for keyboard layouts, so we can drop that godawful C format for more abstract configuration files like those used by MSKLC(although someone would have to write a compiler for that format).
And an element type for each module type, because <module type="..." ... is plain bad XML (how the hell do you validate that beforehand? if you thought "I'll do it in the C++ source!", congratulations! you are part of the problem!). A part from the validation problem we have to be able to use object inheritance when creating new module types. The current situation requires adding a bunch of if's and code duplication.
New build tools required
Some time has to be spent creating a build tool to parse MSKLC .klc files and turn them into C source code. We have 75 modules with almost the same copy-pasted coded with little variations.
Clean separation of ruleset and driver
The Mingw Backend is currently a mess with a lot of copy-pasted code. Rbuild should not contain particular logic for handling some of the modules as it currently has for PSDK, DSDK, mingw_* , debug_sup etc... all this information has to be specified using rbuild syntax in rbuild files. All references to modules by name should be removed from rbuild code.
Compiler- and linker-agnosticism
The information regarding compiler and linker flags should be abstracted to generic elements and translated to the compiler/linker command line switch by the appropriate backend. Not just appended like with our current <linkerflag and <compilerflag. This is vital to support Microsoft tools, and will come in handy for static verification of rbuild build files
Localization of ReactOS
ReactOS localization currently derives from Wine's, which sucks: all languages are linked in all executables, all the time. MUI is now a standard, documented Windows feature, and we should start making use of it, splitting localized executables into a language-neutral (or better, en-US) executable and one satellite resource DLL for each supported language. Doing this automatically and with the least effort possible on the part of coders, is the responsibility of rbuild - or rather, the appropriate helper tool, that rbuild should drive with no human intervention other than a global conditional directive.
Naturally, this will practically disable all localizations until MUI support is added to ReactOS itself... but this feature will have to be catered for by a future redesign of rbuild. One of its effects will be a list of output files (the satellite DLLs) that isn't known with certainty until all executables have been compiled and linked (and all the supported localizations enumerated - because MUI support should happen automatically, regardless of the structure of source code and without the need for any new rbuild directives), complicating the automated creation of a list of files to be added to an installation source.
(Re)factoring the rbuild driver
Not only rbuild as a whole sucks; its core, the driver application written in C++, was developed in somewhat questionable ways, too. Can we fix it without tossing the whole of it out? Let's analyze the many ways it sucks, first
The sorry state of back-ends
Anyone who has had to work with rbuild is familiar with the terrible back-ends, especially the mingw32 one (incidentally, the most frequently used one). Duplicated string literals, non-object-oriented C++, and stuff that really should go in rbuild configuration files
And do we really need back-ends that produce project files? The whole tree certainly cannot be made of project files - Visual Studio doesn't have all tools necessary, and too many dependencies would have to be represented by custom build steps. It could be of some value for "clean" modules - the ones that should be buildable with public SDKs - but it should not be expected to be a viable way to make a Visual C++ build - just a way to make debugging and development easier for users of Visual Studio. Who will ensure the project files generated in this way will stay buildable, though? It would require an official build server running Windows
Isolating the mid-tier - or: rbuild as a library
Instead of plugging in back-ends, the mid-tier itself should be pluggable, to create drivers with a custom front-end and back-end: they would inherit rbuild's logic of parsing rbuild files, but with a different user interface and different outputs. Tools operating on the source tree could be created in this fashion without complicating rbuild's front-end or compiling useless back-ends every time. Examples of such tools include source tree documentation tools, showing modules and their dependencies
Rbuild as a stand-alone project
To ensure rbuild is not tied to the ReactOS tree in any way, and to get rid of the bootstrap makefiles, it should be developed separately, and offered as a separate download and/or bundled with RosBE. To get to this point, rbuild must be improved to the point it can take over the whole build process, including host tools and ISO generation. Ideally, rbuild should be capable of self-hosting, too.
Phase 3: Procrastination
Wherein developers will dust up their ill-earned CS degrees, the patterns book that's currently under the leg of a table to level it, and in general put all their enthusiasm and productivity into postponing the actual work as further as possible. They will analyze the requirements of the current source tree and the proposed new features, evaluate the feasibility of each feature, and produce several hundred lines of text, none of which can actually be compiled into a program.
Please note that, currently, this planning phase is, in turn, still being planned.
Phase 4: The Question
Exactly why does ReactOS have its own build system? If people hate it cannot stuff like scons a broad used tool be considered?
Considering something like scons does 2 things. Number 1 gets improving and maintaining of the build system out of ReactOS problems. Number 2 if there is something wrong with it valuable developer of the ReactOS project does not have to be wasted on fixing it since we can bug report it to the project that made it. Number 3 Problems in the build system is no longer blame one of the ReactOS developers.
For replacement what requirements does the replacement need to do exactly.
