Overview
Examples
Screenshots
Comparisons
Applications
Download
Documentation
Tutorials
Bazaar
Status & Roadmap
FAQ
Authors & License
Forums
Funding Ultimate++
Search on this site
Search in forums












SourceForge.net Logo
Home » Community » U++ community news and announcements » AutoTests
AutoTests [message #42929] Mon, 14 April 2014 20:39 Go to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
I would like to announce that I have finished work on U++ nightly autotesting:

Each night, autotest cron gets started which basically compiles all packages (with handful of exceptions) from examples, reference, tutorial, with CLANG and CLANG in C++11 mode.

It then compiles & RUN all packages from autotest nest - these are supposed to test U++ features that are testable in automatic mode (therefore, it unfortunately excludes GUI).

In total, it is now 357 packages to compile and 73 autotest packages to run.

The results are emailed, for now to me. If you are interested about getting nightly test results, please PM me.

Reasonable additions to autotest are welcome. I also think that something similiar should work for bazaar, I can setup autotest for bazaar on my machine as well, but I would like to keep those separated (and hope somebody else will be responsible for it Smile.

Mirek
Re: AutoTests [message #42932 is a reply to message #42929] Mon, 14 April 2014 22:12 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

That is great news Smile Testing is always good thing to do.

May I suggest using Watchdog to display the results and their history publicly? This is exactly the kind of task that it was written for Wink

Best regards,
Honza
Re: AutoTests [message #42956 is a reply to message #42932] Wed, 16 April 2014 14:34 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
dolik.rce wrote on Mon, 14 April 2014 20:12

May I suggest using Watchdog to display the results and their history publicly? This is exactly the kind of task that it was written for Wink


Does it accept emails? Smile

But generally, yes, I suppose that this info should be further processed. I was thinking

a) new svn 'tested' tag should be created automatically
b) U++ health status should be updated on website - perhaps watchdog based

Mirek
Re: AutoTests [message #42958 is a reply to message #42956] Wed, 16 April 2014 16:34 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

mirek wrote on Wed, 16 April 2014 14:34
dolik.rce wrote on Mon, 14 April 2014 20:12

May I suggest using Watchdog to display the results and their history publicly? This is exactly the kind of task that it was written for Wink


Does it accept emails? Smile

Not yet Smile But it can send them Wink It has a command-line client, that sends the results. But since autotest is U++ app, it might be easier to use the client API directly (I'd just have to make it compile as library and distribute it with the headers).

mirek wrote on Wed, 16 April 2014 14:34
But generally, yes, I suppose that this info should be further processed. I was thinking

a) new svn 'tested' tag should be created automatically
b) U++ health status should be updated on website - perhaps watchdog based

Mirek

Not sure if tested tag is necessary... But displaying some status summary on website would be definitely good idea.

The best thing in watchdog IMHO is that it can put things in perspective. You can see directly how often and how much things break and how long they take to fix.

Also, with watchdog you could test every revision. If you're afraid about abusing too much server CPU, it can run distributed. I think there is many people who would contribute part of their hardware Smile I'd certainly be first one to do so Smile

Anyway, if I have some spare time, I'll try to set-up a publicly accessible instance of watchdog and set it to track upp-mirror and run autotest on my machine. I think that might be best way to convince you Very Happy

Honza
Re: AutoTests [message #42959 is a reply to message #42958] Thu, 17 April 2014 08:13 Go to previous messageGo to next message
mirek is currently offline  mirek
Messages: 13975
Registered: November 2005
Ultimate Member
dolik.rce wrote on Wed, 16 April 2014 14:34

Also, with watchdog you could test every revision. If you're afraid about abusing too much server CPU, it can run distributed. I think there is many people who would contribute part of their hardware Smile I'd certainly be first one to do so Smile


I does not run on website CPU, it runs on nightly build machine... (at my home).

Anyway, single run is about 3 hours, so it is unlikely to be possible to test each revision.

Mirek
Re: AutoTests [message #42961 is a reply to message #42959] Thu, 17 April 2014 09:57 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

mirek wrote on Thu, 17 April 2014 08:13
dolik.rce wrote on Wed, 16 April 2014 14:34

Also, with watchdog you could test every revision. If you're afraid about abusing too much server CPU, it can run distributed. I think there is many people who would contribute part of their hardware Smile I'd certainly be first one to do so Smile


I does not run on website CPU, it runs on nightly build machine... (at my home).

Anyway, single run is about 3 hours, so it is unlikely to be possible to test each revision.

Mirek

Not a problem, with 2 client machines (yours and mine, for example Smile ) you could test ~16 revisions a day, which is probably enough. Of course, if you don't want to have your home pc crunching tests all day, you can set it up to only contribute by testing one revision a day. The client is fully in control regarding what and how much is tested. We could discuss this over a beer, if you're interested Wink

Honza
Re: AutoTests [message #42987 is a reply to message #42929] Mon, 21 April 2014 21:42 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

Hi Mirek,

I've set up watchdog instance that tracks U++ mirror, but now I'm struggling with the AutoTest itself Smile

I managed to compile uppboc/AutoTest, but I haven't found any documentation on how to run it. I see I need a configuration file and a 'test file', whatever it means. What are those and where can I obtain some example that I can use as inspiration?

Honza
Re: AutoTests [message #43001 is a reply to message #42987] Tue, 22 April 2014 17:34 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

dolik.rce wrote on Mon, 21 April 2014 21:42
I managed to compile uppboc/AutoTest, but I haven't found any documentation on how to run it. I see I need a configuration file and a 'test file', whatever it means. What are those and where can I obtain some example that I can use as inspiration?

Never mind, I figured it out from the code... It'll just need few tiny changes to make it work nicely with watchdog.

Honza
Re: AutoTests [message #43003 is a reply to message #42929] Tue, 22 April 2014 21:04 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

It's up and running http://upp-thewatchdog.rhcloud.com/results Very Happy First results are in, for now only for GCC in debug mode, I might add CLANG and/or release mode later...

I'm not sure if I use correct settings, I just run tests for everything in autotest assembly. But that can be always fixed later. I guess this is sufficient for demonstration purposes.

Also, there is a bug with timings, the start times are 6 hours shifted, because the server is in different time zone than the client, so please ignore this till I fix it Wink

Honza

Re: AutoTests [message #43066 is a reply to message #42929] Wed, 30 April 2014 02:20 Go to previous messageGo to next message
wimpie is currently offline  wimpie
Messages: 46
Registered: March 2013
Location: holland
Member
Hi, first of all, nice work Wink

Out of curiosity I just looked at the page, and saw at the GCC test results of apr 28 a lot of "-Wparentheses" warnings
aren´t those Clang warnings?

just mentioning, maybe you are allready on it Wink

grtz

oh and it says "failed 95%"
shouldn that be "failed 5%" or so? (4 fails out of 74)

[Updated on: Wed, 30 April 2014 02:23]

Report message to a moderator

Re: AutoTests [message #43067 is a reply to message #43066] Wed, 30 April 2014 06:23 Go to previous messageGo to next message
dolik.rce is currently offline  dolik.rce
Messages: 1789
Registered: August 2008
Location: Czech Republic
Ultimate Contributor

wimpie wrote on Wed, 30 April 2014 02:20
Hi, first of all, nice work Wink

Out of curiosity I just looked at the page, and saw at the GCC test results of apr 28 a lot of "-Wparentheses" warnings
aren´t those Clang warnings?

just mentioning, maybe you are allready on it Wink

grtz
No, these are from GCC, I have it set up with -Wall parameter in my build method. This part of log is just from building AutoTest itself, perhaps it shouldn't be there at all. It's not really important (as long as it builds correctly) Smile

wimpie wrote on Wed, 30 April 2014 02:20
oh and it says "failed 95%"
shouldn that be "failed 5%" or so? (4 fails out of 74)

You're right, it looks bit misleading. It's actually two pieces of information, "overall status" and "succes rate", concatenated together. It should be visually separated, something like "95% (Failed)" or even something like "95% (F:3, E:1)". If you find more problems, just PM me or create new issue on watchdogs github.

Honza
Re: AutoTests [message #43068 is a reply to message #43067] Wed, 30 April 2014 12:04 Go to previous message
wimpie is currently offline  wimpie
Messages: 46
Registered: March 2013
Location: holland
Member
dolik.rce wrote on Wed, 30 April 2014 06:23
No, these are from GCC, I have it set up with -Wall parameter in my build method. This part of log is just from building AutoTest itself, perhaps it shouldn't be there at all. It's not really important (as long as it builds correctly) Smile

oh ok. never saw it from gcc, first noticed this with clang Wink

dolik.rce wrote on Wed, 30 April 2014 06:23

You're right, it looks bit misleading. It's actually two pieces of information, "overall status" and "succes rate", concatenated together. It should be visually separated, something like "95% (Failed)" or even something like "95% (F:3, E:1)". If you find more problems, just PM me or create new issue on watchdogs github.

Honza

ah I see, overall status is failed. that clarifies

grtz
Previous Topic: Debian repo for nightly builds?
Next Topic: Forum upgraded to 3.0.5
Goto Forum:
  


Current Time: Thu Mar 28 19:59:54 CET 2024

Total time taken to generate the page: 0.01515 seconds