Antivirus Benchmarking v1.0

Anti-virus General Use Benchmarks v1.0

Intentions and Goals

This is sort of inspired by c627627's posts in this thread .

My intention here is a demonstration of the effect various anti-virus products available today have on system performance. There was already an article on the front page regarding their different performances in scanning for virii but none yet have created any benchmarks affected by these applications.

Granted my access to such an extensive array of anti-virus products is far less than that of the author of the front page article. I will do my best to test as many of the "big ones" as possible. The current focus will largely be on symantec products since that is what stemmed this post.

The base for this will be a Windows XP Pro SP2 install onto one clean partition and all drivers asked for by the motherboard. 1 CD ROM, 1 IDE drive, mouse, and KB. (floppy drive was used when imaging the PC and was removed before benching)

My goal here is to determine how much of an impact on system performance various anti-virus programs have. Including memory, CPU and disk performance, as well as boot and shutdown times. However I do not have the ability to infect a computer with viruses, not to mention their existence affects a computer regardless of AV software, and that demonstration has already been done in the front page article mentioned earlier. These tests/benchmarks are for idle everyday use of the software. (realtime protection)

The System

The system that I will be using is a folding machine whose motherboard recently died (the replacement should be here shortly). The specs are as follows. All settings will be at stock, do not want possible instability of OC to affect things. Parts are cheap I know but its just a folding box and it will suffice well enough for this test until perhaps I can repeat the test on my future main rig once I get it. (which is why this is version 1.0)
The PC is:
Intel P4 2.4 CPU
Intel I865GHX motherboard
Pqi 512MBx2 PC3200 RAM
WD1200BB Hard Disk
Generic PSU (500W)

Test Subjects


Norton Anti-virus 2006 (completed)
Norton Anti-virus corp 10 (completed)
Norton Anti-virus corp 8 (completed)
PC Cillin 2006 Internet Security (completed)
PC Cillin 2005 Internet Security (completed)
Macafee Internet Security 2006 (completed)
Kaspersky X.X (completed)
AVG 7.1 (completed)
antivir Workstation X.X (completed)

A note on the antivir programs tested, you will likely notice that certain programs are no longer listed, the most specifics were Norton corp 9, 2005, 2003, and the internet security package for 2006. This is due to the fact that as most of us know Norton products are not free and the older programs are near impossible to find(symantec nearly pretends the older version never existed once the new one comes out). These programs may be included in benchmark 2.0 but we will have to see what I can come up with. After all my personal collection is limited (since you only need one AV program anyways lol) so thats just how it is. The other reason the norton products were omitted is I feel we had a decent spectrum of tests for those based on conclusions made by others regarding corp edition resource hogging and such. So there you have it.


Bench Programs


Super Pi
HD Tach
HD Tune
SiSoft Sandra(CPU arithmetic/memory bandwidth)
PCmark 2005

Other:
Regmon/filemon/tcpview - (10/2/06)These programs will not be used, tried them on a test run and the results were just too much (talking 1000s and 1000s of entries for filemon and several 1000 for regmon)

The Process
How I intend to go about doing this. I will be creating a base install of windows XP, all device drivers and all files needed to install all of the applications. This will then be turned into a ghost image to speed up the process (a lot). I will then, in turn, install one anti-virus product including all its available updates, followed by a few reboots and any pre-scan it wants me to do. I will then install all of the benchmark utilities and run a defrag (trying to eliminate outside effects) then, one at a time run each benchmark and record the results via logs/screen shots. Then start the process over with a different AV product. Since this will be a machine for folding I don't really have a problem with doing this again in the future should more people have questions about other products (perhaps anti-spy ware or other non-anti-virus). We shall see how things go I guess.

Status

After slapping the mobo around a bit, I discovred a bad stick of RAM, which would explain a LOT of things so the base system has changed. And might even be a better test (will now be 2x256 instead of 2x512) for people out there with less RAM. So windows is installing now, more to come soon.
7/18/06 - Gotten everything installed and started benchmarks for base system, Will then be imaging the install and starting in on the testing.
7/19/06 - didnt get much done, was out partying till 1AM Naruto 3+beer shots = ftw
10/2/06 - yes, several months later, I have a new stable build, fully updated OS and updated applications (of versions listed above) and have completed like 95% of the tests. Since I have a special request for NAV 2003 I will be doing this as well when I can find my disc(got waay too many lol). Otherwise assume this benchmark is completed and results will be posted early this week.

Results

Things that are being recorded in addition to the benchmark progs noted above:
Time it takes the machine to boot to login screen, then how much time it takes to login and how long it takes the PC to shutdown.
Teaser: so far there is little performance impact when running AV products.. except!! *cough* but not all of them. Anyways, expect results soon ^.~

Well folks, finally finished, went and downloaded most recent versions of applications and re-ran all tests due to the delay in completing this. As we all know benchmarks can vary slightly so you will see some odd results on most of the SiSandra results but the timer ones (bootup/shutdown/superPi) are some good firm numbers. So without further adue(sp ) here are the results for the programs tested. Perhaps in the future I will do this again with a different system and programs, but for now these are the ones I tested.


















So I think I can draw a pretty good conclusion that most AV programs do not affect the system as much as we may think they do (other than those few obvious exceptions). However, I must admit that these results were not quite what I expected, especially where the memory/cpu/HDD benchmarks are concerned. To find that these programs really affect the machine so little on day to day use is honestly a shock to me.

2 interesting follow up points. When running benchmarks the Trendmicro 2006 edition (PCCillin) was the only product to warn me about the benching programs, thinking they might be unsafe to run. The other point is out of all of the products Kaspersky was the only one that went and did things on it's own without me telling it to. With the machine just sitting there Kaspersky went out, downloaded updates and installed them without even asking for a confirmation nor having me setup auto updates. Some consider this a good thing, others don't.


*edit* software added
*edit* more software
*edit* another section, and more software)
*edit* more software and verified versions/years more (and grammar/spelling)
*edit* more changes/updates, benchmark almost completed.
*edit* added results and updated various benchmark information
*edit* was reorganizing photobucket stuffs and broke image links, now they fixed

avast has a free version (home edition)
clamwin is a opensource av for windows
and comodo.com has a free Antivirus application

if your interested

Avast can be free i think.

If this is done right, i think it can answer alot of questions for people on performance, as all know everyone bashes norton for being slow.

i found Avast to be a resource hog, kapersky 6 i use now and i dont notice it.

perhaps Filemon and Regmon can be used to see what files the AV access behind the scenes? If it is reading files it doesnt need to ?

You have a private message, pik4chu.

In addition to the champ: AntiVir 7, it would be nice to see the results for Norton 2003, the last Norton product that requires no activation and reportedly the last one not described as a resource hog.


The runner up for actually finding virii was:
Trend Micro PC-cillin Internet Security 2005/2006

Then came
Norton Antivirus 2005/2006
McAfee Internet Security Suite 2006
Kaspersky Anti-Virus 5/6

All others did not make it into top 5 obviously, it would be nice to see if this is still true.

People post about other Antivirus programs but they did not make it into the top 5 according to an independent test done by one of our members.


The old chart of his results I made for personal use, it was not published:

i still find it very interesting how the online scan finds more then an installed program.

Mr.Guvernment said:

i still find it very interesting how the online scan finds more then an installed program.

Click to expand...

maybe its cause the online ones see thier installed counterpart for the virus that it is and the offline ones don't count themselves?

and there are only two cases of this anyways bitdefender witha difference of 5? pfft and kaspersky, with a much bigger difference but I question the versions used.

I like the way the tests were done. The guy took a computer box and infected it with thousands of virii, every package he could find on every underground internet virus sight. Then he did the tests.


Now here's what I say to every person questioning the results or telling me 'such & such' antivirus program is better:

1. Do the same thing that guy did, I'll help you, just PM me and let's look at the results if they are different.
2. AntiVir found and continues to find nasties on machines I test. When it does so, I try to install a competing program to see if it can detect them, most on the list cannot, certainly the lower rated ones do not and we're not talking about exotic viruses either.


I have no personal attachment to any antivirus program, but that's what it's all about: my machine having a virus and most of the progs not detecting them.

AntiVir detects them, Norton does not for example.


How many viruses (all right, virii) can it take to mess your system up real good?

One.

Nod32 has a free trial, and it's FAST when it comes to scanning. I'd recommend giving that a try, if you would.

...even though 75% of tested programs performed better than Nod32 when it came to virus detection.

How many viruses (all right, virii) can it take to mess your system up real good?

One.

Click to expand...

Yeah, but there's something to be said for likelihood as well. I personally use AVG, and while it sucks at detecting more exotic virii...
1) Exotic virii are less likely to be contracted - there's a reason they're exotic
2) Exotic virii will be generally lower-risk - there's a reason they're exotic.



I'm interested in NOD32 because that's what I use to scan computers I fix for people. Catches a few less virii, but generally cleans the system much faster. Anything high-risk should be included in the definitions for all of these, anyway.

*EDIT* Hmm, this thread is all white/blue/green.

johan851 said:

Yeah, but there's something to be said for likelihood as well. I personally use AVG, and while it sucks at detecting more exotic virii...
1) Exotic virii are less likely to be contracted - there's a reason they're exotic
2) Exotic virii will be generally lower-risk - there's a reason they're exotic.



I'm interested in NOD32 because that's what I use to scan computers I fix for people. Catches a few less virii, but generally cleans the system much faster. Anything high-risk should be included in the definitions for all of these, anyway.

*EDIT* Hmm, this thread is all white/blue/green.

Click to expand...

its allg reen/white/blue cause only true geeks are concerned about system benchmarks in regards to different AV products

after some heat issues and such windows is installing, weee.

Installing bench progs and eventually windows updates (all except WGA thank you very much)

johan851 said:

*EDIT* Hmm, this thread is all white/blue/green.

Click to expand...

Then let me chime in with my opinion to ruin the white/blue/green thread =)

pik4chu, I think that this is a great project that you are undertaking. It will be great to see how much an anti-virus program truly affects your system performance. If I may suggest, you could take a look at --This Sticky-- to get a hold of a few more free anti-virus programs. You might also be able to get a hold of free trials of certain anti-virii programs, and test those as well. I look forward to seeing how things turn out.

Two things that I would like to see tested would be boot up and shut down times. By recording the time it takes to boot from when u push the button, to the log-on screen, to actually being able to run a program, and from the time you hit the shut down button, to when your computer actually shuts down. I think that it would be good to know how each anti-virus program affects these aspects, as many people constantly spend money to upgrade their systems to boot faster. I personally have had quite a few people come to me asking which part of their systems they could change to increase boot times, (especially individuals with systems still running older PII / PIII rigs with 256megs of RAM). Since these individuals are usually not computer enthusiasts, it is hard to explain to them how to look through 'services' and their start ups, to cut down on unnecessary programs. It would be great to recommend to them an Anti-Virus program other than Norton & McAfee (which is what most of them use) with proof to back up my claim that there are other programs that are a) Free, b) are just as good at catching virii, and c) decrease boot time.

In any case, I will be checking this thread on a daily basis, and looking forward to the preliminary results! If you need any help with anything at all, just hit me a PM, and I'll do my best to help out.


Raven

and they're off!!
baseline benchmarks completed, image created and first test "victim" is under way/ antivir is the first to be tried. And since I can reimage the machine in <5min things should move along much faster than anticipated.

Also, filemon and regmon log a disgusting amount of data so I wont be making that data pretty or anything for the final notes unless someone specifically wants them (talking log files of thousands of lines each).

I hope to have this all done by the end of the weekend.

ohhh, why diddnt i see this thread before?
i would LOVE to see the results.
particurly the impact as things are read from/written to the harddrive with many of the "allways on" protection scemes that sit between the O/S and the hardware playing traffic cop.
weekly scan times or whatever are still good to know, but not as important.

finally somebody getting down to buisness !!!


cant wait for results ....

pik4chu, we're going to make an article for the front page out of this. I hope you don't mind.

Should your results verify that Norton is a system resource hog galore, I hope Silversinksam's ritual sacrifice of that Symantec CD can be worked into it as well.

@)%(@!$)% WQ)#T@!% *********************
finished antivir and during restore the drive im restoring for took a dump and took the other with it lol...get to start all over, atleast I only did one. -_- So I guess stand by for an update.

but for a teaser. antivir didnt hardly do a dang thing to performance. aside from one or two things im talking like .01% changes lol.. weee. anyways, hopefuly things wont be a pain from here on.

and c627627 I dont mind at all, I can make two versions, one thats thread friendly (summaries and links and such) and one for the main page where its all written out. We can talk more about it when I get done.

Yes, Norton Ghost sucks. I just emailed you a nice Drive Image bootable CD I made.

(Drive Image was so much better than Ghost, Symantec bought it, then promptly killed it to eliminate this best imaging product ever made a couple of years back.)

Anyway, set your first Boot device in BIOS to CD, and try it...

Use "Low" compression (not "High" or "None") for the fastest & easiest imaging experience... good luck.

c627627 said:

Yes, Norton Ghost sucks. I just emailed you a nice Drive Image bootable CD I made.

(Drive Image was so much better than Ghost, Symantec bought it, then promptly killed to eliminate this best imaging product ever made a couple of years back.)

Anyway, set your first Boot device in BIOS to CD, and try it...

Use "Low" compression (not "High" or "None") for the fastest & easiest imaging experience... good luck.

Click to expand...

Actually, I did 2 test restores just to verify things worked and all was well, imaging took jus over 2 minutes and I was back in biz, but it turns out Asus doesn't actually know how to fix a god damn thing so it was hardware related. The problem was the motherboard caused the image to fail, which killed the install on the main drive, then I couldnt even see the second hard drive, then bootup time started taking a minute longer so GG Asus hope you enjoy another RMA, losers. Redoing this on a completely different machine that I know works fine (been running F@H for 2 weeks now) Will update specs accordingly later. But will def check out the prog you sent.

c627627 said:

...even though 75% of tested programs performed better than Nod32 when it came to virus detection.

Click to expand...

its funny cause all i see people advertising around here is how GREAT NOD32 is! obviously they didnt read the benchmark thread.