Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » U++ community news and announcements » Universal makefile & UppBuilder
Universal makefile & UppBuilder [message #35381] Wed, 08 February 2012 18:59 Go to next message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

Hi everyone!

A long post follows. If you are lazy to read, just scroll down for the short version Wink

Almost exactly two years ago, I got inspired by extraordinary features of GNU make to create universal makefile, that could be used to build U++ apps without TheIDE. This beast then evolved and was rewritten few times. It was used used for building packages for ubuntu, and it worked well enough. But few days ago, there came Max and his improvements to gdb interface and broke the builds (no offense Max, it's great work Wink ). The reason is that the universal makefile does not support including .brc files. So it is time to take next step in evolution and reveal a new makefile, that I have worked on for quite some time. I didn't want to publish it just yet, as some of the features are not 100% finished and some are even missing, but it builds TheIDE, even with .brc files, so it will save us for now. It can probably be called a beta version.

The biggest difference from the old one is that it does not rely on GNU make features. I tried as hard as possible to keep everything POSIX compatible, so that BSD make and others work with it as well. This resulted in complete change in philosophy, instead of some macro-magic parsing upp files there is now C++ parser embeded in the makefile. It is stored in compressed form and extracted and compiled upon first usage. This parser then reads and analyzes the upp files and creates temporary makefile, that builds the application.

Now good news for fans of other build systems than make: the parser is written with those in mind. The code can be found in uppbox/UppBuilder and whoever is interested can try to write an alternative backend that would generate something else than Makefile, e.g. cmake files or scons script. It would be even possible to create a backend that builds the application instead of just describing how to build it Smile One strange thing about the parser is that it is NOT written in U++. There are two reasons for this: it must be simple to build and it must be fast to be build. The former is achieved by having all the code in single file and relying on minimum of dependencies (no libs, only 6 standard headers). The later requirement forced me to use almost no templates (only 1 class and 1 function) and compiling without optimizations. The result is ~15kB of code, that is compiled in less than 1s on any modern hardware (tested on my lazy Intel Atom netbook;) ), that can parse all the packages necessary to build TheIDE and generate the makefile in about 0.2s. This is definitely an improvement, compared to ~20s that it took with the old makefile Wink Nice bonus is that the code is IMHO quite well documented, as well as it's usage.

Here are the major changes from the old makefile:
  • C++ parser is used instead of make+bash magic
  • Better POSIX compatibility and hence greater portability
  • Faster... much faster
  • Similar, but slightly improved interface (the make variables)
  • Handles brc files
  • Can perform some custom build steps (experimental)
  • Interactive mode: user can be asked for some details, e.g. what configuration to use
  • The file counter eye-candy was dropped (it was often broken anyways)
  • Colorized output - new eye-candy Smile

You can see the generated makefile in uppbox/lpbuild2/mkfile, if you don't want to examine the UppBuilder too closely.

TL;DR version:
There is new, better universal makefile here, if you used the old one you might want to switch.
If you intend to build U++ sources using other build systems than TheIDE or make, you might want to check new UppBuilder package.

Best regards,
Honza
Re: Universal makefile & UppBuilder [message #35382 is a reply to message #35381] Wed, 08 February 2012 22:23 Go to previous messageGo to next message
Mindtraveller is currently offline  Mindtraveller
Messages: 917
Registered: August 2007
Location: Russia, Moscow rgn.
Experienced Contributor

Very cool!
Did you test its compatibility with *BSD?
Re: Universal makefile & UppBuilder [message #35384 is a reply to message #35382] Wed, 08 February 2012 23:24 Go to previous messageGo to next message
Didier is currently offline  Didier
Messages: 671
Registered: November 2008
Location: France
Contributor
Hi Honza,

I'm quite impressed, you went all this way just to enhance UPP build wow Shocked
But managing all the portability issues with the make facilities is a pretty tough shot !

I've been thinking for some while that boost-build is very well adapted to Upp package hierarchy and it works on WIN/LINUX/BSD/MAC/.... everywhere there is a decent C compiler.

boost-build is a build tool that has it's own syntax ( quite simple ) and that manages package dependencies (as in Upp) and it supports many compilers on many OS.
It is written in C ==> if you have a C compiler/linker and you can build it and then use it to build whatever you wan't ==> you don't depend on presence of make/gmake/imake/xxtools/...whatever.

On big gain is that the same "makefile" works on all platforms !! and that most common compilation options are available in generic form (optimisations, 32/64 bits, ...)

Re: Universal makefile & UppBuilder [message #35385 is a reply to message #35382] Wed, 08 February 2012 23:33 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

Mindtraveller wrote on Wed, 08 February 2012 22:23

Very cool!
Did you test its compatibility with *BSD?

I did tests on PCBSD in early stages of development, but I didn't have time to test it properly in this final form. So there might be some errors on BSD, but probably easy to fix.

Honza
Re: Universal makefile & UppBuilder [message #35386 is a reply to message #35384] Wed, 08 February 2012 23:44 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

Didier wrote on Wed, 08 February 2012 23:24

Hi Honza,

I'm quite impressed, you went all this way just to enhance UPP build wow Shocked
It's my long-term project, I like to see how I can improve on the same topic as my skills are leveling up Smile Also, I'm actively using it to build packages in launchpad, to avoid chicken-egg problem with TheIDE, so that is a good motivation.
Didier wrote on Wed, 08 February 2012 23:24

But managing all the portability issues with the make facilities is a pretty tough shot !
I tried to use only documented, POSIX compatible features, so as long as the tools on other platforms follow POSIX, it should work. Some of the POSIX specifications are linked in documentation, if you're interested Wink

Didier wrote on Wed, 08 February 2012 23:24

I've been thinking for some while that boost-build is very well adapted to Upp package hierarchy and it works on WIN/LINUX/BSD/MAC/.... everywhere there is a decent C compiler.

boost-build is a build tool that has it's own syntax ( quite simple ) and that manages package dependencies (as in Upp) and it supports many compilers on many OS.
It is written in C ==> if you have a C compiler/linker and you can build it and then use it to build whatever you wan't ==> you don't depend on presence of make/gmake/imake/xxtools/...whatever.

On big gain is that the same "makefile" works on all platforms !! and that most common compilation options are available in generic form (optimisations, 32/64 bits, ...)

Hi Didier,
I've never seen boost-build in action, but I believe that if necessary I could almost match it's portability just by writing another backend in some common language. It should be possible to implement the builds in shell script or even in C and avoid make completely Smile But for now, I believe that supporting systems with make is enough. When the code is stable and bug-less, we can play with other possibilities Smile

Honza
Re: Universal makefile & UppBuilder [message #35387 is a reply to message #35386] Wed, 08 February 2012 23:59 Go to previous messageGo to next message
Didier is currently offline  Didier
Messages: 671
Registered: November 2008
Location: France
Contributor
Quote:

I've never seen boost-build in action


Well I tried it out a few times mostly at home, but I used it once at work and it saved my ass when I had to turn over the project to another person in another company... which only had a linux station with no make tool installed (and on which it was forbidden to install anything !!!!)

So I just recompiled boost-build, added the 32bit compiler option (the new linux station was 64bits while mine was 32bits) and the app was running fine again.
Re: Universal makefile & UppBuilder [message #35392 is a reply to message #35387] Thu, 09 February 2012 07:35 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

Didier wrote on Wed, 08 February 2012 23:59

Quote:

I've never seen boost-build in action


Well I tried it out a few times mostly at home, but I used it once at work and it saved my ass when I had to turn over the project to another person in another company... which only had a linux station with no make tool installed (and on which it was forbidden to install anything !!!!)

So I just recompiled boost-build, added the 32bit compiler option (the new linux station was 64bits while mine was 32bits) and the app was running fine again.


Ok, so I've just looked at it and it looks as a nice tool. As long as it is installed on the targeted system. It is binary tool, just as make, so there is not much difference. The portability is nice, but I still think that POSIX compatible shell backend in UppBuilder could be even more portable Smile But it is becoming kind of off-topic, I just wanted to announce that there is a new makefile, if anyone wants to use it Wink

Honza
Re: Universal makefile & UppBuilder [message #35393 is a reply to message #35392] Thu, 09 February 2012 08:05 Go to previous messageGo to next message
unodgs is currently offline  unodgs
Messages: 1363
Registered: November 2005
Location: Poland
Ultimate Contributor

That's great you put so much effort to get working upp packages on arch linux! But I have a one question. Wouldn't be better to simply compile umk first and then use it to build theide?
Re: Universal makefile & UppBuilder [message #35394 is a reply to message #35393] Thu, 09 February 2012 08:38 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

unodgs wrote on Thu, 09 February 2012 08:05

That's great you put so much effort to get working upp packages on arch linux! But I have a one question. Wouldn't be better to simply compile umk first and then use it to build theide?

How could I compile umk, if you don't have umk to compile it with? Smile

Umk consists of 200+ source files, that require correct flags to be set and correct libraries to be linked. OTOH UppBuilder parser is single file (actually there is 2, but they are merged for the purpose of makefile), which doesn't depend on any flags nor libraries. That simplifies things a lot Wink

Honza

PS: Also, last time I checked, the U++ web said that "if the reinvented wheel spins about 4 times faster, reinventing is not that bad idea..." Very Happy

[Updated on: Thu, 09 February 2012 08:42]

Report message to a moderator

Re: Universal makefile & UppBuilder [message #35395 is a reply to message #35394] Thu, 09 February 2012 08:49 Go to previous messageGo to next message
unodgs is currently offline  unodgs
Messages: 1363
Registered: November 2005
Location: Poland
Ultimate Contributor

dolik.rce wrote on Thu, 09 February 2012 02:38


Umk consists of 200+ source files, that require correct flags to be set and correct libraries to be linked. OTOH UppBuilder parser is single file (actually there is 2, but they are merged for the purpose of makefile), which doesn't depend on any flags nor libraries. That simplifies things a lot Wink


Ok, that's explain a lot.. I thought that recently separated umk is much simpler to compile Smile

Re: Universal makefile & UppBuilder [message #35423 is a reply to message #35381] Sat, 11 February 2012 21:11 Go to previous messageGo to next message
chickenk is currently offline  chickenk
Messages: 168
Registered: May 2007
Location: Grenoble, France
Experienced Member
Great work Jan!

Just a very small suggestion while I look at the code: it would be interesting to add the '-v' option to hexdump in brc.sh ; I let you read the manual page to understand. Smile

I got stuck with the same problem with waf (i.e. brc files), and I'll try to reuse parts of your new framework, of course!

Cheers
Lionel
Re: Universal makefile & UppBuilder [message #35427 is a reply to message #35423] Sun, 12 February 2012 12:18 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

chickenk wrote on Sat, 11 February 2012 21:11

Just a very small suggestion while I look at the code: it would be interesting to add the '-v' option to hexdump in brc.sh ; I let you read the manual page to understand. Smile
Good catch Lionel, hexdump -v is definitely good idea to prevent trouble Smile

Thanks,
Honza
Re: Universal makefile & UppBuilder [message #37222 is a reply to message #35381] Thu, 06 September 2012 22:49 Go to previous messageGo to next message
sam_ is currently offline  sam_
Messages: 4
Registered: July 2009
Location: Pazrava pri Puchove
Junior Member
Hi Jan

I've got small bug report for you. On Linux Mint 13 Xfce 64bit (Ubuntu 12.04 based) makefile from trunk r5321 fails:
sam@herkales ~/.upp_makefile $  make PKG=Bombs FLAGS="GCC"
/bin/sh: 5: Syntax error: "then" unexpected (expecting "done")
make: *** [do-build] Error 2

This is caused by /bin/sh linking to /bin/dash. After
sam@herkales ~/.upp_makefile $ sudo rm /bin/sh
[sudo] password for sam: 
sam@herkales ~/.upp_makefile $ sudo ln -s /bin/bash /bin/sh

makefile works like a charm.

Thanks for your excellent work.
Lubos
Re: Universal makefile & UppBuilder [message #37225 is a reply to message #37222] Fri, 07 September 2012 09:54 Go to previous message
dolik.rce is currently offline  dolik.rce
Messages: 1781
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

Thank for report Lubos.

I'll try to fix that as soon as I get home from vacation. Note that you can also force another shell by executing make like this
make SHELL=/bin/bash PKG=...


Best regards,
Honza
Previous Topic: TheIDE startup should be now faster
Next Topic: Skylark "new package" template
Goto Forum:
  


Current Time: Mon Sep 27 00:00:23 CEST 2021

Total time taken to generate the page: 0.01126 seconds