Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
Home
Discussion Groups
DB Engine
SQL ServerMSDESQL Server CE
Services
Analysis (Data Mining)Analysis (OLAP)DTSIntegration ServicesNotification ServicesReporting Services
Programming
CLRConnectivitySQLXML
Other Technologies
ClusteringEnglish QueryFull-Text SearchReplicationService Broker
General
Data WarehousingPerformanceSecuritySetupSQL Server ToolsOther SQL Server Topics
DirectoryUser Groups
Related Topics
MS AccessOther DB ProductsMS Server Products.NET DevelopmentVB DevelopmentJava DevelopmentMore Topics ...

SQL Server Forum / DB Engine / SQL Server / March 2008

Tip: Looking for answers? Try searching our database.

SQL Server file defragmentation

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Jim - 10 Mar 2008 22:21 GMT
Hello, I have a SQL Server 2000 box (Server 2003 sp1) that I have been
running a scheduled task to defrag the drives using the built-in defrag.  It
appears that this built-in defrag is not keeping up or doing its job very
well.

I was planning on abandoning the on-board defrag routine in favor of Raxco's
Perfect Disk defragmentation software and was wondering if there were any
particular files that should be excluded from defragmentation software.
I've googled and haven't found much other than it's a good idea to run some
form of defrag software.

Any defrag best practices I should be aware of?

Thanks
Jim
Andrew J. Kelly - 10 Mar 2008 23:57 GMT
The SQL Files should only have to be defragged once in their lifetime. The
only way they would get fragmented again is if you constantly grow and
shrink them. Set them to the size you need them to be for the next 6 months
to a year, defrag them and you should be done.

Signature

Andrew J. Kelly    SQL MVP
Solid Quality Mentors

> Hello, I have a SQL Server 2000 box (Server 2003 sp1) that I have been
> running a scheduled task to defrag the drives using the built-in defrag.
[quoted text clipped - 11 lines]
> Thanks
> Jim
Jim - 11 Mar 2008 00:19 GMT
We have databases that the vendor says we shouldn't touch or they will not
support - so I'm reluctant to change any settings on them should we ever
need tech support.

Are you saying that it is unadvisable to run a defrag program on a SQL
server or that one shouldn't need to if the sizes are set correctly?

Thanks
Jim

> The SQL Files should only have to be defragged once in their lifetime. The
> only way they would get fragmented again is if you constantly grow and
[quoted text clipped - 16 lines]
>> Thanks
>> Jim
Linchi Shea - 11 Mar 2008 02:10 GMT
I'd question whether there is any real value in running the filesystem-level
defrag regularly. Just because some defrag vendors out there scare people
doesn't mean you should do so or do so regularly. If you can, try to quantify
the benefits. For instance, you can pick some sample queries or maintenance
routines, and evaluate their performance before and after your file defrag to
see if you have reaped any real benefit.

Linchi

> We have databases that the vendor says we shouldn't touch or they will not
> support - so I'm reluctant to change any settings on them should we ever
[quoted text clipped - 26 lines]
> >> Thanks
> >> Jim
Andrew J. Kelly - 11 Mar 2008 02:24 GMT
File level defrag tools such as DiskKeeper etc are for defragmenting at the
OS or file level and have nothing what so ever to do with SQL Server.  The
fragmentation that may be an issue is the fragmentation within the file
caused by page splits etc. DiskKeeper will not help with fragmentation
within the files. The way to remove that fragmentation is with proper index
maintenance. If the vendor denies support because you changed the file size
you probably should be looking for a new vendor.

Signature

Andrew J. Kelly    SQL MVP
Solid Quality Mentors

> We have databases that the vendor says we shouldn't touch or they will not
> support - so I'm reluctant to change any settings on them should we ever
[quoted text clipped - 26 lines]
>>> Thanks
>>> Jim
MartinWilliams0817 - 11 Mar 2008 18:59 GMT
Hi Jim,
If I may be permitted to give you an answer on this?
FYI, I've worked as a SQL Server Production DBA for 12+ years,so have
encountered this situation many, many times.

Database fragmentation happens at two levels. Period :

1. Within the database files. The mdfs, ldfs, ndfs or whatever you
name their extensions. Badly fragmented database files  WILL, 100%
guaranteed affect your database's performance.
The database reads and writes data to a file in the file system. If
that file is spread across the disk badly enough, you will get bad
performance as the disk head has to move around the disk more to read
the data, thrashing itself to death. It's that simple. This is disk
file fragmentation.

2. Within the database itself, at the page level. This means that the
index pages get out of order within the .mdf database file. The file
itself on disk may be one single, contiguous block, but if the index
pages are jumbled up and spread out within that file, it will also
affect database performance. This is index fragmentation.

If you have one or the other you *may* get a performance problem. If
you have both badly enough, you definitely will have one.

But first of all, establish if you really do need to do anything. If
your users are happy with their response times, and the database isn't
slowing to a halt, then there may not be any need to do anything,
though preventative maintenance is always the preferred option. If
things are creaking at the seams, Linchi's advice on baselining your
server is very wise. No point in regularly defragging for six hours
each night if you get no benefit !  Do some simple user measurements -
stopwatch the time from entering a zip code to customer data coming
back, ten times before and after the defrag. Throw away the fastest
and slowest and take an average for each and compare.

My advice would be:
First, don't defrag your disk files in the day time when users are
using the database, or during critical periods at night (eg: during
backup operations, batch jobs etc).
Lots of conflicting disk I/O between SQL Server and the disk defrag
job means timeouts, extended backup Windows and irate users.
Personally, I'd schedule it in for once a month max. If as Andrew
rightly said you've sized your database *files* so that they're not
always growing, they shouldn't get heavily fragmented at the disk
level too quickly.

Second, the built in Windows defragmenter is better than nothing, but
not great.
I've not used Raxco's, but heard it is a very good product. Likewise
the server edition of Diskeeper (note that the Windows defragger is a
cut down version of the Diskeeper product, but the latter has more
options).
Each will argue the virtues of these products - your choice.

Third, most third party vendors don't size their databases correctly,
either initially or for growth, so you typically have a scenario where
a busy db (eg a customer call centre db) is extending itself by 10Mb
every few hours/days. Not good. It is in this scenario where you will
find database fragmentation at the file level. But your vendor says
don't change the size, which *may* be ridiculous, depending on the
current database size and how regularly it is growing. The db will
have to either autogrow as data is added to the database and the
current maximum file size is reached, or the database will come to a
halt when the database file(s) are full and you won't have a usable
system. The former is the more likely scenario, but unless they've
chosen a large growth size, a busy database will fragment badly and
quickly, requiring a disk level defrag. If it's small and relatively
unused, no need to worry so much. If it grows frequently, then your
database app vendor needs educating. If not, they may have a valid
point, but it's worth addressing as some vendors are plain lazy or
lacking database skills.

Fourth, if you decide you need to defrag your hard drives, then always
defrag your SQL Server indexes AFTER the hard drive defrag. If you do
it the other way around, you risk re-distributing your nicely defrag'd
indexes randomly across the disk surface. I've seen database system
where this has been done in the wrong order and they get slower rather
than quicker. Always disk defrag then index defrag, if you're going to
do a disk defrag at all. With regard to defrag products, if you're not
versed in T-SQL commands or can't face the tedium of writing your own
scripts / debugging someone else's, you can plump for a tool such as
Visual Defrag or SQL Defrag Manager, which both ease the pain of
getting the job done and make things quicker in a large environment
with loads of servers and databases. But always use them AFTER you've
done the disk defrag, I can't stress that enough.

Fifth, with regard to defrag frequency, it depends on numerous
factors. With disk files, how large your database is sized, how
quickly it grows, the growth rate of your business's databases, and
the frequency of inserts, deletes and updates all play a part. There
is no simple answer to this, it all depends on the usage profile of
your particular database. But Andrew's advice to manually grow the
database so that it's good for the next 6-12 months is good, so long
as you can predict your growth rates. With regard to index defrags, if
your database is very dynamic, you'll be re-indexing every night.
Quieter databases, once a week/month or less. It all depends.

Finally, consider defragging your non database files one night and
your database files (mdfs, ldfs) the next, and repeating over several
nights, at least initially. In a badly fragmented system with large
drives holding a lot of data this will cut down your defrag time into
two chunks. This can be important if your overnight maintenance window
is small. Depending on the product and the amount of disk space, you
may have to do this 2-3 times before you file are optimally organised.
Alternatively, schedule the job to run over a public holiday starting
the previous evening, if this is possible. And make sure there's
15-20% free disk space before the disk defrag starts.

Whatever, if you're wise you'll always ensure that you've got backups
before you reorganise a database's index pages. Never lost one yet as
a result of a disk or an index defrag, but there's always a first
time....
Martin Williams.
davidkellis32@googlemail.com - 14 Mar 2008 13:57 GMT
Hi Jim,
If I may be permitted to give you an answer on this?
FYI, I've worked as a SQL Server Production DBA for 12+ years,so have
encountered this situation many, many times.

Database fragmentation happens at two levels :

1. Within the database files. The mdfs, ldfs, ndfs or whatever you
name their extensions. Badly fragmented database files will, 100%
guaranteed affect your database's performance.
The database reads and writes data to a file in the file system. If
that file is spread across the disk badly enough, you will get bad
performance as the disk head has to physically move across the surface
of the disk more to read the data, thrashing itself to death. This is
disk file fragmentation.

2. Within the database itself, at the data page level. Here, the index
pages get out of order within the .mdf database file. The file itself
on disk may be one single, contiguous block, but if the index pages
are jumbled up and spread out within that file, it will also affect
database performance. This is index fragmentation.

If you have one or the other you *may* get a performance problem. If
you have both badly enough, you definitely will have one.

But first of all, establish if you really do need to do anything. If
your users are happy with their response times, and the database isn't
slowing to a halt, then there may not be any need to do anything,
though preventative maintenance is always the preferred option. If
things are creaking at the seams, Linchi's advice on baselining your
server is very wise. No point in regularly defragging for six hours
each night if you get no benefit !  Do some simple user measurements
at the front end. Simply stopwatch the time from entering a zip code
to customer data coming back, ten times before and after the defrag.
Throw away the fastest and slowest and take an average for before and
after the initial defrag, and compare.

Other advice I'd recommend would be:

First, don't defrag your disk files in the day time when users are
using the database, or during critical periods at night (eg: during
backup operations, batch jobs etc).
Lots of conflicting disk I/O between SQL Server and the disk defrag
job means timeouts, extended backup Windows and irate users.
Personally, I'd schedule it in for once a month max. If as Andrew
rightly said you've sized your database *files* so that they're not
always growing, they shouldn't get heavily fragmented at the disk
level too quickly.

Second, the built in Windows defragmenter is better than nothing, but
not great.
I've not used Raxco's, but have heard it is a very good product.
Likewise the server edition of Diskeeper (note that the Windows
defragger is a cut down version of the Diskeeper product, but the
latter has more options).
Each will argue the virtues of these products - your choice.

Third, most third party vendors don't size their databases correctly,
either initially or for growth, so you typically have a scenario where
a busy db (eg a customer call centre db) is extending itself by 10Mb
every few hours/days. Not good. It is in this scenario where you will
find database fragmentation at the file level. But your vendor says
don't change the size, which *may* be ridiculous, depending on the
current database size and how regularly it is growing. The db will
have to either autogrow as data is added to the database and the
current maximum file size is reached, or the database will come to a
halt when the database file(s) are full and you won't have a usable
system. The former is the more likely scenario, but unless they've
chosen a large growth size, a busy database will fragment badly and
quickly, requiring a disk level defrag. If it's small and relatively
unused, no need to worry so much. If it grows frequently, then your
database app vendor needs educating. If not, they may have a valid
point, but it's worth addressing as some vendors are plain lazy or
lacking database skills.

Fourth, if you decide you need to defrag your hard drives, then always
defrag your SQL Server indexes AFTER the hard drive defrag. If you do
it the other way around, you risk re-distributing your nicely defrag'd
indexes randomly across the disk surface. I've seen database system
where this has been done in the wrong order and they get slower rather
than quicker. Always disk defrag then index defrag, if you're going to
do a disk defrag at all. With regard to defrag products, if you're not
versed in T-SQL commands or can't face the tedium of writing your own
scripts / debugging someone else's, you can plump for a tool such as
Visual Defrag or SQL Defrag Manager, which both ease the pain of
getting the job done and make things quicker in a large environment
with loads of servers and databases. But always use them AFTER you've
done the disk defrag, I can't stress that enough.

Fifth, with regard to defrag frequency, it depends on numerous
factors. With disk files, how large your database is sized, how
quickly it grows, the growth rate of your business's databases, and
the frequency of inserts, deletes and updates all play a part. There
is no simple answer to this, it all depends on the usage profile of
your particular database. But Andrew's advice to manually grow the
database so that it's good for the next 6-12 months is good, so long
as you can predict your growth rates. With regard to index defrags, if
your database is very dynamic, you'll be re-indexing every night.
Quieter databases, once a week/month or less. It all depends.

Finally, consider defragging your non database files one night and
your database files (mdfs, ldfs) the next, and repeating over several
nights, at least initially. In a badly fragmented system with large
drives holding a lot of data this will cut down your defrag time into
two chunks. This can be important if your overnight maintenance window
is small. Depending on the product and the amount of disk space, you
may have to do this 2-3 times before you file are optimally organised.
Alternatively, schedule the job to run over a public holiday starting
the previous evening, if this is possible. And make sure there's
15-20% free disk space before the disk defrag starts.

Whatever, if you're wise you'll always ensure that you've got backups
before you reorganise a database's index pages. Never lost one yet as
a result of a disk or an index defrag, but there's always a first
time....

Dave Ellis.
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.