Home » Community » U++ community news and announcements » AutoTests
| AutoTests [message #42929] |
Mon, 14 April 2014 20:39  |
 |
mirek
Messages: 14255
Registered: November 2005
|
Ultimate Member |
|
|
I would like to announce that I have finished work on U++ nightly autotesting:
Each night, autotest cron gets started which basically compiles all packages (with handful of exceptions) from examples, reference, tutorial, with CLANG and CLANG in C++11 mode.
It then compiles & RUN all packages from autotest nest - these are supposed to test U++ features that are testable in automatic mode (therefore, it unfortunately excludes GUI).
In total, it is now 357 packages to compile and 73 autotest packages to run.
The results are emailed, for now to me. If you are interested about getting nightly test results, please PM me.
Reasonable additions to autotest are welcome. I also think that something similiar should work for bazaar, I can setup autotest for bazaar on my machine as well, but I would like to keep those separated (and hope somebody else will be responsible for it  .
Mirek
|
|
|
|
| Re: AutoTests [message #42932 is a reply to message #42929] |
Mon, 14 April 2014 22:12  |
|
|
That is great news Testing is always good thing to do.
May I suggest using Watchdog to display the results and their history publicly? This is exactly the kind of task that it was written for 
Best regards,
Honza
|
|
|
|
|
|
| Re: AutoTests [message #42958 is a reply to message #42956] |
Wed, 16 April 2014 16:34 |
|
|
mirek wrote on Wed, 16 April 2014 14:34dolik.rce wrote on Mon, 14 April 2014 20:12
May I suggest using Watchdog to display the results and their history publicly? This is exactly the kind of task that it was written for
Does it accept emails?
Not yet But it can send them It has a command-line client, that sends the results. But since autotest is U++ app, it might be easier to use the client API directly (I'd just have to make it compile as library and distribute it with the headers).
mirek wrote on Wed, 16 April 2014 14:34But generally, yes, I suppose that this info should be further processed. I was thinking
a) new svn 'tested' tag should be created automatically
b) U++ health status should be updated on website - perhaps watchdog based
Mirek
Not sure if tested tag is necessary... But displaying some status summary on website would be definitely good idea.
The best thing in watchdog IMHO is that it can put things in perspective. You can see directly how often and how much things break and how long they take to fix.
Also, with watchdog you could test every revision. If you're afraid about abusing too much server CPU, it can run distributed. I think there is many people who would contribute part of their hardware I'd certainly be first one to do so
Anyway, if I have some spare time, I'll try to set-up a publicly accessible instance of watchdog and set it to track upp-mirror and run autotest on my machine. I think that might be best way to convince you 
Honza
|
|
|
|
|
|
| Re: AutoTests [message #42961 is a reply to message #42959] |
Thu, 17 April 2014 09:57 |
|
|
mirek wrote on Thu, 17 April 2014 08:13dolik.rce wrote on Wed, 16 April 2014 14:34
Also, with watchdog you could test every revision. If you're afraid about abusing too much server CPU, it can run distributed. I think there is many people who would contribute part of their hardware I'd certainly be first one to do so
I does not run on website CPU, it runs on nightly build machine... (at my home).
Anyway, single run is about 3 hours, so it is unlikely to be possible to test each revision.
Mirek
Not a problem, with 2 client machines (yours and mine, for example ) you could test ~16 revisions a day, which is probably enough. Of course, if you don't want to have your home pc crunching tests all day, you can set it up to only contribute by testing one revision a day. The client is fully in control regarding what and how much is tested. We could discuss this over a beer, if you're interested
Honza
|
|
|
|
| Re: AutoTests [message #42987 is a reply to message #42929] |
Mon, 21 April 2014 21:42 |
|
|
Hi Mirek,
I've set up watchdog instance that tracks U++ mirror, but now I'm struggling with the AutoTest itself
I managed to compile uppboc/AutoTest, but I haven't found any documentation on how to run it. I see I need a configuration file and a 'test file', whatever it means. What are those and where can I obtain some example that I can use as inspiration?
Honza
|
|
|
|
| Re: AutoTests [message #43001 is a reply to message #42987] |
Tue, 22 April 2014 17:34 |
|
|
dolik.rce wrote on Mon, 21 April 2014 21:42I managed to compile uppboc/AutoTest, but I haven't found any documentation on how to run it. I see I need a configuration file and a 'test file', whatever it means. What are those and where can I obtain some example that I can use as inspiration?
Never mind, I figured it out from the code... It'll just need few tiny changes to make it work nicely with watchdog.
Honza
|
|
|
|
| Re: AutoTests [message #43003 is a reply to message #42929] |
Tue, 22 April 2014 21:04 |
|
|
It's up and running http://upp-thewatchdog.rhcloud.com/results First results are in, for now only for GCC in debug mode, I might add CLANG and/or release mode later...
I'm not sure if I use correct settings, I just run tests for everything in autotest assembly. But that can be always fixed later. I guess this is sufficient for demonstration purposes.
Also, there is a bug with timings, the start times are 6 hours shifted, because the server is in different time zone than the client, so please ignore this till I fix it
Honza
|
|
|
|
|
|
| Re: AutoTests [message #43067 is a reply to message #43066] |
Wed, 30 April 2014 06:23 |
|
|
wimpie wrote on Wed, 30 April 2014 02:20Hi, first of all, nice work
Out of curiosity I just looked at the page, and saw at the GCC test results of apr 28 a lot of "-Wparentheses" warnings
aren´t those Clang warnings?
just mentioning, maybe you are allready on it
grtz No, these are from GCC, I have it set up with -Wall parameter in my build method. This part of log is just from building AutoTest itself, perhaps it shouldn't be there at all. It's not really important (as long as it builds correctly)
wimpie wrote on Wed, 30 April 2014 02:20oh and it says "failed 95%"
shouldn that be "failed 5%" or so? (4 fails out of 74)
You're right, it looks bit misleading. It's actually two pieces of information, "overall status" and "succes rate", concatenated together. It should be visually separated, something like "95% (Failed)" or even something like "95% (F:3, E:1)". If you find more problems, just PM me or create new issue on watchdogs github.
Honza
|
|
|
|
| Re: AutoTests [message #43068 is a reply to message #43067] |
Wed, 30 April 2014 12:04 |
wimpie
Messages: 46
Registered: March 2013
Location: holland
|
Member |
|
|
dolik.rce wrote on Wed, 30 April 2014 06:23 No, these are from GCC, I have it set up with -Wall parameter in my build method. This part of log is just from building AutoTest itself, perhaps it shouldn't be there at all. It's not really important (as long as it builds correctly)
oh ok. never saw it from gcc, first noticed this with clang
dolik.rce wrote on Wed, 30 April 2014 06:23
You're right, it looks bit misleading. It's actually two pieces of information, "overall status" and "succes rate", concatenated together. It should be visually separated, something like "95% (Failed)" or even something like "95% (F:3, E:1)". If you find more problems, just PM me or create new issue on watchdogs github.
Honza
ah I see, overall status is failed. that clarifies
grtz
|
|
|
|
Goto Forum:
Current Time: Sat Apr 26 15:09:45 CEST 2025
Total time taken to generate the page: 0.01744 seconds
|
Members
Pages
Help
Register
Login
Home
