Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » Coffee corner » The power of Makefile
icon3.gif  The power of Makefile [message #24947] Fri, 05 February 2010 15:45 Go to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

Hello everybody!

I just have to share this bit of kowledge I've just gained with all of you... Until yesterday, I thought that make is just another simple unix tool. But after reading Managing projects with GNU make, I changed my mind.

Actually make is more like a programming language. It provides a lot of features and possibilities I would ever think of. Like defining functions, built-in rules, pattern expansion and many more.

Just to show you (at least to those of you who don't know make really good): Following Makefile is just a few lines, but it can analyze U++ package dependencies. For now, it is restricted to one directory, so it works only for uppsrc or in directory with exported project. Also, it ignores flags (just includes everything in "uses") . Both of this could be probably fixed with few more lines. Another problem might be it's speed. For ide it takes about 12s on my laptop, but for average-sized packages, it's less than second (also it's definitely not optimal, some steps are duplicated). Ok, here is the code:
PKG=ide
DEP=$(PKG)

define get-upp
 $(if $(findstring /,$1),$1/$(notdir $1).upp,$1/$1.upp)
endef

define get-uses
 $(strip $(shell cat $(call get-upp,$1) | tr ",\n;" "  \n" | grep "uses" | sed 's/(.*)//'))
endef

define get-deps
 $(sort $1 $(foreach d,$(sort $(subst \,/,$(filter-out uses%,$(call get-uses,$1)))),$(call get-deps,$d)))
endef

.PHONY: all

all:
	@echo "To build $(PKG) following packages are needed:"
	@echo "$(call get-deps,$(PKG))"
If you want to try it, just create file uppsrc/Makefile and copy this in. To execute call make or make PKG=AnyOtherPackage. Actually this is what I like best - the universality. Actually I believe that it would be possible to extend this simple example to not only analyze, but also compile any package. The only information that would have to be added would be assembly structure, flags and compiler/linker options (or path to .bm and .var file).

If you read all the way down here, thanks for your patience Wink

Best regards,
Honza
Re: The power of Makefile [message #24949 is a reply to message #24947] Fri, 05 February 2010 16:04 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 12311
Registered: November 2005
Ultimate Member
Does it work with BSD-make too?

Mirek
Re: The power of Makefile [message #24950 is a reply to message #24949] Fri, 05 February 2010 16:32 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
luzr wrote on Fri, 05 February 2010 16:04

Does it work with BSD-make too?

Unfortunately this does not work with BSD make.
It does not support function definitions and it also does not have build in functions like 'findstring'.

Matthias
Re: The power of Makefile [message #24983 is a reply to message #24950] Sat, 06 February 2010 20:39 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

masu wrote on Fri, 05 February 2010 16:32

luzr wrote on Fri, 05 February 2010 16:04

Does it work with BSD-make too?

Unfortunately this does not work with BSD make.
It does not support function definitions and it also does not have build in functions like 'findstring'.

Matthias


It should work with gmake. From what I read, it is fairly standard on BSD. I think some ports even require gmake to build properly...

Bye,
Honza
Re: The power of Makefile [message #24990 is a reply to message #24983] Sat, 06 February 2010 22:46 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
Hi Honza,

It works with gmake (GNU make) which by the way is used to build TheIDE from ports at the moment.
So you are right, concerning the make utility it wouldn't change anything.
But BSD make itself is a different utility (the default make on BSD systems).

Matthias
Re: The power of Makefile [message #24991 is a reply to message #24990] Sat, 06 February 2010 22:55 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
I tried your Makefile with gmake on FreeBSD and it works fine Smile.

Matthias

[Updated on: Sat, 06 February 2010 22:56]

Report message to a moderator

Re: The power of Makefile [message #24999 is a reply to message #24991] Sat, 06 February 2010 23:34 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

masu wrote on Sat, 06 February 2010 22:55

I tried your Makefile with gmake on FreeBSD and it works fine Smile.

Matthias

Hi Matthias,

Thanks for testing! I was just playing with this idea, so I was not concerned about compatibility. After looking into BSD make docs, I believe I could do the same in BSD style too. It would just contain more shell commands. And it wouldn't be compatible with GNU make Smile BSD make uses different syntax for conditional inclusions. So if we want to use more of the makes power, we should stick to GNU version as it is available pretty much anywhere, from Linux through BSD and mac to cygwin.

Now, does someone actually think (like me Smile ) that this is fascinating? Having universal Makefile for any project is tempting idea, even though I'm not sure if it would be really useful.

Honza
Re: The power of Makefile [message #25001 is a reply to message #24947] Sun, 07 February 2010 01:42 Go to previous messageGo to next message
mr_ped is currently offline  mr_ped
Messages: 805
Registered: November 2005
Location: Czech Republic - Praha
Experienced Contributor

I like GNU make a lot too. Usually when I see somebody whining about make and asking for scons or something else, it turns out he has no idea how to use make properly and it would suit him perfectly.

That said, make is not final solution, in some cases there are better tools (for building upp applications I strongly prefer TheIDE Smile), but many people bashing make don't have idea...
Re: The power of Makefile [message #25220 is a reply to message #25001] Sun, 14 February 2010 21:08 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
Hi,

Like I mentioned here:
http://www.ultimatepp.org/forum/index.php?t=msg&th=4968& amp; amp;start=0&
I took up Honza's tests with GNU make and I would like to extend the idea to have a Makefile for building arbitrary U++ packages.

The version I have at the moment is supporting packages that are located including all its prerequisite packages. It does not support building packages distributed over several assemblies, but for now the Makefile should be put into uppsrc assembly and can then be used to build TheIDE.

I have tested it with FreeBSD and Ubuntu. I have also tried it in OpenBSD, but there seem to be other problems not related to the make process. At least all parameters (i.e. source files, link libraries) were extracted correctly on that system, too.

Here is a checklist I would like to work on:

1. Add support for building from other directories (to only provide on Makefile in root directoy for source releases)
2. Add support for setting object files' output directory
3. Add support for package building over different assemblies

And of course any error fixing in the meantime if erros occur during testing.

Edit: I forgot to mention, that per default (if you just type make) TheIDE is build with GTK bindings and shared linking.
Tip: You van also build with make -jx where x is the number of parallel processes run by make. I used parameter -j2 on my Dual Core system.

Matthias
  • Attachment: Makefile.tbz
    (Size: 1.89KB, Downloaded 215 times)

[Updated on: Sun, 14 February 2010 21:14]

Report message to a moderator

icon14.gif  Re: The power of Makefile [message #25223 is a reply to message #25220] Sun, 14 February 2010 22:21 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

Hi Matthias,

You've got my deep respect! I didn't expect anyone even trying my ideas, let alone doing some work. But you almost finished it and actually much faster than I expected that it would take. Very impressive!

I'm just testing the Makefile and trying to figure out how it works. I have problems to even find the parts that I wrote Smile But it seems to be working perfectly. Theide compiled perfectly on my machine, so I guess there should be no problems with other packages. And by the way, thanks for the hint with -j parameter for parallel builds, I didn't know it is that easy.

Now about the Makefile. First of all, I love the way you made it verbose. Reporting every step is really user friendly. I would even suggest (optional) hiding of the commands, so only the commands would be visible. About your TODO list: Numbers 1 and 2 should be easy. Just adding a variable with path on proper places. I'll try to figure it out. Number 3 is trickier, but most important. Currently, only uppsrc packages can be built, since other standard assemblies span across two directories. Hopefully we will find a way how to search through both.

Oh, and I almost forgot, another great think about this, that was not possible with the exported Makefiles: if you put it in directory with all sources, you can build any number of (main) packages to producing several binaries, while effectively caching the .o files. Similar way as theide does it. I think this is the biggest feature we have got here. I guess we could even add one more cycle and build multiple binaries at once with something like make "PKG=SocketClient SocketServer". Something like this is quite difficult to do in theide...

I would be very happy if this Makefile could be once distributed with U++ as an alternative to current exported Makefiles. I'm aware that it will never supersede it, because of it's complexity and also because the exported files are BSDmake compatible. This one just looks better to me. Even if just because it compresses 37000 lines into 150 Smile It is really elegant.

Best regards,
Honza
Re: The power of Makefile [message #25252 is a reply to message #25223] Mon, 15 February 2010 16:24 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
Hi Honza,

first, thanks for your appreciation Smile.

A problem I have is to find a way to define variables when calling make which can be extended then during the make flow.
I thought VAR+=x should work, but it does not work for env variables defined as parameter to make, like this:

make DEFS=-DflagNOGTK

In this case the variable DEFS is completely overwritten by -DflagNOGTK. But what I want is to add this flag to already defined flags inside the Makefile.

Do you have an idea?

Matthias
Re: The power of Makefile [message #25253 is a reply to message #25252] Mon, 15 February 2010 16:50 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

masu wrote on Mon, 15 February 2010 16:24

Hi Honza,

first, thanks for your appreciation Smile.

A problem I have is to find a way to define variables when calling make which can be extended then during the make flow.
I thought VAR+=x should work, but it does not work for env variables defined as parameter to make, like this:

make DEFS=-DflagNOGTK

In this case the variable DEFS is completely overwritten by -DflagNOGTK. But what I want is to add this flag to already defined flags inside the Makefile.

Do you have an idea?

Matthias

Hi,

This is usually solved using another variable:
FIXEDFLAGS=...
FLAGS=$(FIXEDFLAGS) $(DEFS)
.

Simple, stupid, but works Smile

Honza

Re: The power of Makefile [message #25316 is a reply to message #25253] Tue, 16 February 2010 23:24 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
Hi,

here is the latest version Honza and I made.
It should be possible to build any desired package in U++ root dir Smile. If it is not a main package use APP= on the cmd line (see below).
It was not extensively tested, so expect some glitches Wink.

The following variables (with defaults in []) are supported on the cmd line:
  • PKG: package name to build [ide]
  • APPNAME: name of output program file [$(PKG).out]
  • ASSEMBLY: list of directories searched for packages [uppsrc]
  • INCLUDES: extra include flags [too long]
  • LDFLAGS: extra link flags [-L/usr/X11R6/lib -L/usr/lib -L/usr/local/lib]
  • DEFS: list of additional U++ flags [GCC]
  • OBJDIR: output directory for object files and libraries [_out]
  • VERBOSE: verbose stdout [off]

Link libraries and mainconfig flags are extracted.
Object files and libraries are put into output directories separated by package flag configuration (similar to how TheIDE it does).

Examples:

TheIDE(SHARED): make
TheIDE(SHARED NOGTK): make DEFS=NOGTK
TheIDE(NOGTK): make DEFS=NOGTK SHARED=
Controls4U_Demo(SHARED): make PKG=Controls4U_Demo ASSEMBLY=bazaar

Matthias

  • Attachment: Makefile.tbz
    (Size: 2.36KB, Downloaded 200 times)

[Updated on: Wed, 17 February 2010 16:05]

Report message to a moderator

Re: The power of Makefile [message #25841 is a reply to message #24947] Mon, 15 March 2010 08:57 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

Hello all!

After a long time I finally got to back to the Makefile... and I finally finished everything I wanted to.

So the file I present you here is rewritten from scratch. It is faster and smarter. New features are:
  • Compiles multiple packages at once
  • Reads and uses most of the options from .upp in the same way as theide
  • Verbosity switches
  • Simulation mode
  • Help target
  • Commented code (a bit) Wink
  • Several bugs fixed
  • Little ASCII Art bonus Wink

The last thing I miss is blitz Smile
BTW: There are even some options that can not be set in theide Smile Or at least I couldn't find how... E.g. optimize file/package for size.

To get started, just copy the file into upp directory. I advise you to start with typing
make
#OR
make help
Both show you a help text, that contains a description and default values for variables that can controll the build process.

If you want to try something more elaborated, you can test something like this:
make "PKG=ide usvn `ls examples reference tutorial |sed 's/[^ ]*://g' |tr '\n' ' '`" USEMAINCFG=y BINDIR=bin/ BINEXT= JOBS=3
This will compile theide, usvn and all the packages in mentioned folders, using flags from first line in mainconfig and put the binaries into a bin subdirectory. All the compiling will use 3 parallel jobs to speed the things up. Expect some error messages, as some of the packages are win only, have errors in code or have non-working mainconfig.

I tested the makefile by building more than 200 applications with various flags and randomly running many of them, without any encountering any errors in the makefile. So I believe that it should work in most cases. If anyone tries this, please let me know about your experiences Wink Also, if you have any questions, feel free to ask (on forum or IRC).

Best regards,
Honza
  • Attachment: Makefile
    (Size: 19.07KB, Downloaded 220 times)
Re: The power of Makefile [message #25842 is a reply to message #25841] Mon, 15 March 2010 11:38 Go to previous messageGo to next message
chickenk is currently offline  chickenk
Messages: 168
Registered: May 2007
Location: Grenoble, France
Experienced Member
Hi Honza,

I tried it, it is absolutely great. Thank you very much for this effort, much appreciated.

Just for record, I added a very small rule 'showflags' (patch attached) that displays the flags for the main package as they are guessed. Example:

$ make PKG=ide showflags
ide_FLAGS: GCC LINUX MAIN
$ make PKG=ide USEMAINCFG=y showflags
ide_FLAGS: GCC GUI LINUX MAIN

Can be useful to know these flags before actually trying to compile.

Another way to do that without the patch:

$ make PKG=ide USEMAINCFG=y -p | grep '^ide_FLAGS'
ide_FLAGS := GCC GUI LINUX MAIN


Just my 2 cents. Keep up this good work, will surely help u++ to expand.
Re: The power of Makefile [message #25843 is a reply to message #25841] Mon, 15 March 2010 12:13 Go to previous messageGo to next message
koldo is currently offline  koldo
Messages: 3096
Registered: August 2008
Ultimate Member
Hello Honza

It looks great. What should they be the next steps to integrate this and Launchpad into U++ infrastructure ?


Best regards
IƱaki
Re: The power of Makefile [message #25845 is a reply to message #25841] Mon, 15 March 2010 12:31 Go to previous messageGo to next message
masu is currently offline  masu
Messages: 378
Registered: February 2006
Senior Member
Hi Honza,

thank you for continuing the effort, great job Very Happy !

I propose to discard uppsrc/ide/Makefile for future releases and include Honza's Makefile in root dir.
Either we define defaults to build TheIDE or we also provide a wrapper Makefile to call Honza's with needed variables.

Matthias
Re: The power of Makefile [message #25846 is a reply to message #25842] Mon, 15 March 2010 13:02 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

chickenk wrote on Mon, 15 March 2010 11:38

Hi Honza,

I tried it, it is absolutely great. Thank you very much for this effort, much appreciated.

Just for record, I added a very small rule 'showflags' (patch attached) that displays the flags for the main package as they are guessed. Example:

$ make PKG=ide showflags
ide_FLAGS: GCC LINUX MAIN
$ make PKG=ide USEMAINCFG=y showflags
ide_FLAGS: GCC GUI LINUX MAIN

Can be useful to know these flags before actually trying to compile.

Another way to do that without the patch:

$ make PKG=ide USEMAINCFG=y -p | grep '^ide_FLAGS'
ide_FLAGS := GCC GUI LINUX MAIN


Just my 2 cents. Keep up this good work, will surely help u++ to expand.


Thank you, chickenk!

If you think this is helpful, I will add it. Usability is my main goal Wink Also in future I plan (optional) interactive mode, where user could choose which set of flags from mainconfig to use and some other tricks...

Regards,
Honza
Re: The power of Makefile [message #25847 is a reply to message #25843] Mon, 15 March 2010 13:12 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

koldo wrote on Mon, 15 March 2010 12:13

Hello Honza

It looks great. What should they be the next steps to integrate this and Launchpad into U++ infrastructure ?


Hi Koldo!
This has a little common with Launchpad. I'm not sure why the nightly builds doesn't work yet, probably Mirek is busy... But yes, I'm thinking about using this instead of generated makefiles, it is actually as easy as overwriting one file and changing few lines of script. I'll have a look at it soon.

About integration into U++: This is easy too. Just include the file in builds and mention it in documentation. Additionally someone might integrate it into theide. Probably in similar style as current makefiles, just copying this makefile into export directory and write correct defaults in it.

Regards,
Honza
Re: The power of Makefile [message #25892 is a reply to message #24947] Thu, 18 March 2010 00:18 Go to previous messageGo to previous message
dolik.rce is currently offline  dolik.rce
Messages: 1771
Registered: August 2008
Location: Czech Republic
Senior Contributor

Hello all!

New version is here! Smile Changelog follows:
  • Fixed bug when running via -f option (thanks Matthias)
  • Modified output to make Chickenk more happy with SIMULATE mode output
  • Added export target
  • Expanded help
  • Various little bugs fixed

The most important change is the export target. It parses the makefile with given options and saves it to a file given in EXPORT variable. The resulting file is faster (especially if you work with FAST=n), simpler, smaller, BSD-make compatibleand doesn't require any commandline parameters, since they are all hardcoded in it. You can sure magine now what can it be used for Wink The only downside is that it is a bit messy inside, due to the nature of the exporting process and I don't think that it could be fixed (at least not easily).

Example:
cd ~/uppsvn
make -f ~/Makefile PKG=ide "FLAGS=GUI SSE2 MT SPEED" FAST=n BIN=bin EXPORT=Makefile.ide
#later, as many times you want, possibly on different machine(s) (within same directory tree)
make -f Makefile.ide

Best regards,
Honza
  • Attachment: Makefile
    (Size: 21.35KB, Downloaded 215 times)
Previous Topic: Anybody still knowing anyone using Win98?
Next Topic: What do you think I should do?
Goto Forum:
  


Current Time: Sat Apr 04 11:22:07 CEST 2020

Total time taken to generate the page: 0.01084 seconds