Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
Home
Discussion Groups
DB Engine
SQL ServerMSDESQL Server CE
Services
Analysis (Data Mining)Analysis (OLAP)DTSIntegration ServicesNotification ServicesReporting Services
Programming
CLRConnectivitySQLXML
Other Technologies
ClusteringEnglish QueryFull-Text SearchReplicationService Broker
General
Data WarehousingPerformanceSecuritySetupSQL Server ToolsOther SQL Server Topics
DirectoryUser Groups
Related Topics
MS AccessOther DB ProductsMS Server Products.NET DevelopmentVB DevelopmentJava DevelopmentMore Topics ...

SQL Server Forum / DB Engine / SQL Server CE / October 2007

Tip: Looking for answers? Try searching our database.

Is it just me or are there BIG problems with SQLCE 3.0?

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
An o' Neamus - 04 Oct 2007 07:52 GMT
Greetings,

I wrote a very simple C# console application which was designed to
give us a rough idea on the performance and reliability of SQLCE (the
source code without metrics has been posted to this group recently).
What I've found is that there are major performance and reliability
issues with SQLCE 3.0 as opposed to SQLCE 2.0. Whereas SQLCE 2.0 seems
to be fairly consistent in terms of it's memory usage and the amount
of time taken to insert rows into the table regardless of the size of
the table (to a limit, after about 300,000 records it too has major
performance issues), SQLCE 3.0 seems to slow down noticably and
continually use more memory even after a few thousand records are
inserted and actually crashes (consistently) after 25592 records are
inserted!

What gives? Based on this it would seem that SQLCE 2.0 is far
preferable for an unattended device that needs an uptime of over 30
days whilst constantly (several times per minute) inserting into the
database.

Thanks for your thoughts on this...
José Joye - 04 Oct 2007 20:50 GMT
Looking at the code you posted previously, I have seen that your keep
creating new strings in your Insert loop (500'000).
This is really a memory killer habit. I do not know if this is the problem.
However, try to look at this article for the best way to insert
a larger amount of rows under CE

http://www.pocketpcdn.com/articles/articles.php?&atb.set(c_id)=74&atb.set(a_id)=
11003&atb.perform(details
)=&

Hope this help
- José

> Greetings,
>
[quoted text clipped - 17 lines]
>
> Thanks for your thoughts on this...
joker - 04 Oct 2007 23:06 GMT
A bad habit perhaps, but not the problem.

Even when creating a single string the problems still exists.
An o' Neamus - 05 Oct 2007 08:03 GMT
On Oct 4, 8:50 pm, Jos? Joye
<jose.joye__KILLTHESPAMS__SMAPSEHTLLI...@bluewin.ch> wrote:
> Looking at the code you posted previously, I have seen that your keep
> creating new strings in your Insert loop (500'000).
[quoted text clipped - 3 lines]
>
> http://www.pocketpcdn.com/articles/articles.php?&atb.set(c_id)=74&atb...

Thanks. The actual application that this was a test for will run
continuously and needs to stay up for at least a month. The test is a
simulation of several single inserts that will be done per minute (of
course I didn't actually want to wait a month, so I sped it up). So
it's a valid test IMO because we will not be doing bulk inserts. The
problem is not memory related, as (a) it fails in exactly the same
place on multiple devices with varying memory sizes (and in some cases
over 10Mb of program memory remaining), and (b) it doesn't fail on an
older version of SQLCE (2.0). It's a bug in SQLCE 3.0, which we are
not going to try and work around because if we find a serious bug with
a simple program, then we are not going to take the risk of finding
more bugs with more complex programs.

So to rephrase the question: As there is at least one show-stopper
(for us) bug in SQLCE 3.0 that doesn't appear to be in SQLCE 2.0, as
well as SQLCE 2.0 appearing to have vastly superior performance. Is
there any reason why we would not want to be using SQLCE 2.0?
An o' Neamus - 05 Oct 2007 15:30 GMT
Well, it would appear that this is definitely a SQLCE 3.0 problem.
I've rewritten the same test in C++ using OLE DB, so the only constant
is the choice of underlying database engine. It falls over (ie.
0xc0000005 exception) after inserting the exact same number of
records! Using the SQLCE 2.0 engine with the C++ OLE DB program works
as expected (ie. it works).

Also, taking the average time to insert a record (including flushes
etc.), the SQLCE 2.0 engine appears to be between 3 and 10x faster
than the SQLCE 3.0 engine (depending on the number of records in the
table). The SQL CE 2.0 engine takes a constant amount of time
regardless of the table size, whereas the SQL CE 3.0 engine gets
progressively slower the more records there are in the table. The C++
OLE DB program appears to be between 100 and 200x faster than the .NET
CF C# application when both are running against the SQLCE 2.0 engine
(obviously the .NET CF application could be optimised to reduce the
performance differential).

So it's a slam-dunk decision for us: OLE DB w/ C++ and SQLCE 2.0. Is
there any advantage (to offset the poor performance and reliability)
of using SQL CE 3.0 (assuming we don't need any of the new features)?
Or is there a major disadvantage to using the SQL CE 2.0 engine? We do
not have any need or intention to synchronise our database with a
'real' SQL server.

TIA
William Vaughn - 06 Oct 2007 18:19 GMT
One trick you might try is to keep a connection to the database open. If you
use the Open update Close technique, the database is compressed at the final
close. Keeping a connection open postpones this operation.

Signature

____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant, Dad, Grandpa
Microsoft MVP
INETA Speaker
www.betav.com
www.betav.com/blog/billva
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------

> Greetings,
>
[quoted text clipped - 17 lines]
>
> Thanks for your thoughts on this...
An o' Neamus - 08 Oct 2007 07:27 GMT
> One trick you might try is to keep a connection to the database open. If you
> use the Open update Close technique, the database is compressed at the final
> close. Keeping a connection open postpones this operation.

Thanks. The database was kept open for the entire test run. Another
poster (in this or another thread I started on this subject),
suggested closing the database occasionally to get around the problem
we were seeing (ie. the database engine causing a GPF). As SQL CE 2.0
doesn't crash and is much faster, we've elected to use this version as
opposed to SQL CE 3.0.

At this point, the only unanswered question we have is whether there
are any reasons at all (given that performance and reliability are
severely degraded) why we should even consider using SQL CE 3.0?

TIA
Erik Ejlskov - 08 Oct 2007 16:20 GMT
Ordinary support for SQL Server 2000 (and with that SQL CE 2.0) ends in
april 2008.

Signature

Erik Ejlskov Jensen
- see my SQL Compact blog at
http://erikej.blogspot.com

>> One trick you might try is to keep a connection to the database open. If
>> you
[quoted text clipped - 14 lines]
>
> TIA
William Vaughn - 08 Oct 2007 20:00 GMT
I encourage you to get up on MSDN Connect and start pounding on the SQL CE
3.0 team to help them repro the performance and crash cases you have
experienced. I have passed your comments to the dev team directly but did
not get much more than "It seems to run faster for us...". Since they have
not shipped I think it would make sense for everyone to get these issues
resolved.

hth

Signature

____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant, Dad, Grandpa
Microsoft MVP
INETA Speaker
www.betav.com
www.betav.com/blog/billva
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------

>> One trick you might try is to keep a connection to the database open. If
>> you
[quoted text clipped - 14 lines]
>
> TIA
Ginny Caughey [MVP] - 08 Oct 2007 20:04 GMT
Hi Bill,

It would also be important to have a repro example to test against 3.5 as
well.

Signature

Ginny Caughey
Device Application Development MVP

>I encourage you to get up on MSDN Connect and start pounding on the SQL CE
>3.0 team to help them repro the performance and crash cases you have
[quoted text clipped - 23 lines]
>>
>> TIA
An o' Neamus - 09 Oct 2007 08:55 GMT
> I encourage you to get up on MSDN Connect and start pounding on the SQL CE
> 3.0 team to help them repro the performance and crash cases you have
> experienced. I have passed your comments to the dev team directly but did
> not get much more than "It seems to run faster for us...". Since they have
> not shipped I think it would make sense for everyone to get these issues
> resolved.

Thanks. I've sent a message to the team through their blog as was
requested on the forums.microsoft SQLCE forum. What is MSDN Connect,
and how do I use this to 'pound' the SQL team? :)

I'm including a simplified example below for reference. Please note
that the content of the example is not that important, this same test
has been done with parameterised SQL, SqlCeRecordset and C++/OLE DB.
It *always* fails at exactly the same point (ie. 25592 records), and
never fails on SQLCE 2.0 (obviously the SqlCeRecordset test could not
be done on SQLCE 2.0). As suggested by another poster, I've added a
bit of code into my test which closes and reopens the database
connection after 10000 inserts. This does solve the problem on SQLCE
3.0 (with all of the above access methods). However, I've read that
opening and closing the DB connection is not a good idea. Will this
introduce any additional problems, and if so, what are they?

TIA

Source code:

using System.Data;
using System.Data.SqlServerCe;

namespace TestDB
{
   class Program
   {
       static void Main(string[] args)
       {
           SqlCeConnection sqlConnection = new SqlCeConnection();
           SqlCeCommand sqlCommand = sqlConnection.CreateCommand();
           SqlCeResultSet sqlResult;

           string strDatabase = "\\TestDB.sdf";
           string strTable = "TestTable";

           try
           {
               // Database connection string
               sqlConnection.ConnectionString =
                   "Data Source=" + strDatabase;

               // Database file doesn't exist?
               if (!System.IO.File.Exists( strDatabase ))
               {

                   Console.WriteLine("Creating database: " +
strDatabase);

                   // Create SQL engine object
                   SqlCeEngine sqlEngine = new SqlCeEngine(
                       sqlConnection.ConnectionString);

                   // Use it to create database

                   sqlEngine.CreateDatabase();
               }

               Console.WriteLine("Opening database: " + strDatabase);

               // Open database connection
               sqlConnection.Open();

               // Display SQL version
               Console.WriteLine("SQL Version: "
                   + sqlConnection.ServerVersion.ToString());

               // Check if table exists
               sqlCommand.CommandText = "SELECT TABLE_NAME FROM "
                   + "INFORMATION_SCHEMA.TABLES WHERE "
                   + "TABLE_NAME = '" + strTable + "'";
               sqlResult = sqlCommand.ExecuteResultSet(
                   ResultSetOptions.Insensitive);

               // Table doesn't exist?
               if (!sqlResult.Read())
               {
                   Console.WriteLine("Creating table: " + strTable);

                   // Create table
                   sqlCommand.CommandText = "CREATE TABLE " +
strTable
                       + "(Sequence integer IDENTITY(1,1) NOT NULL
PRIMARY KEY, "
                       + "Timestamp datetime DEFAULT GETDATE(),"
                       + "Message nvarchar(80))";
                   sqlCommand.ExecuteNonQuery();
               }

               string strText;

               for (int i = 1; i <= 500000; i++)
               {
                   Console.Write("\rInserted " + i.ToString() + "
records         ");

                   strText = "Row Number: " + i.ToString();

                   sqlCommand.CommandText = "INSERT INTO " + strTable
                       + " (Message) VALUES ('" + strText + "')";
                   sqlCommand.ExecuteNonQuery();
               }
           }
           // SQL error
           catch(SqlCeException sqlex)
           {
               // Display all error messages
               Console.WriteLine( "ERROR:" );
               foreach (SqlCeError sqlError in sqlex.Errors)
               {
                   Console.WriteLine( sqlError );
               }
           }
           // Other errors
           catch(Exception ex)
           {
               // Display error message
               Console.WriteLine( "ERROR:" );
               Console.WriteLine( ex.Message );
           }
           finally
           {
               // Close connection
               if (sqlConnection.State != ConnectionState.Closed)
               {
                   Console.WriteLine("Closing database");

                   sqlConnection.Close();
               }
           }

           Console.ReadLine();
       }
   }
}
William Vaughn - 10 Oct 2007 20:32 GMT
Connect is the feedback mechanism use to submit bugs to the dev teams. It's
exposed in the help menus in Visual Studio.

Signature

____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant, Dad, Grandpa
Microsoft MVP
INETA Speaker
www.betav.com
www.betav.com/blog/billva
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
Visit www.hitchhikerguides.net to get more information on my latest book:
Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition)
and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook)
-----------------------------------------------------------------------------------------------------------------------

>> I encourage you to get up on MSDN Connect and start pounding on the SQL
>> CE
[quoted text clipped - 140 lines]
>    }
> }
An o' Neamus - 09 Oct 2007 18:12 GMT
> One trick you might try is to keep a connection to the database open. If you
> use the Open update Close technique, the database is compressed at the final
> close. Keeping a connection open postpones this operation.

Thanks. This doesn't appear to improve performance much (if at all).
What does happen though, is that the amount of free memory on the
device gradually decreases until it's about 240Kb. It doesn't
completely run out, with 500Kb being freed before it gets a chance to
completely run out. If you then do something else on the device that
(in combination with the test program) causes the device memory to
drop to about 140Kb, then (I guess it panics) and 1Mb gets freed.
Rinse and repeat.

Same device, same program, one connection, closing the connection
often, obviously slows it down, but keeps the available memory around
6Mb.

Is this a random example that causes a need to close and reopen the
connection (to avoid a GPF)? Is the general rule to keep one
connection open for the life of the program? Will adding complexity to
the program and tables possibly change the environment such that the
GPF might not happen (this example is obviously much less complex than
the actual application's database needs)? Should we design the
application so that it occasionally closes and reopens the database
(bearing in mind it has to run indefinitely, and at least for 30
days)?

TIA
An o' Neamus - 16 Oct 2007 14:37 GMT
> Is this a random example that causes a need to close and reopen the
> connection (to avoid a GPF)?

After some contact with the SQL CE team, it appears that this (GPF)
problem is in fact a random example. The solution to the problem is to
specify a value for the columns that have DEFAULT values (in this case
the TimeStamp column), even if the value specified is the same as the
DEFAULT (definitely a bug, but with a simple workaround).

It did take a fair amount of prodding to get the SQL CE team to
provide a workaround. First, no response, next "we can't duplicate
this", then "this is not a problem in the new (unreleased) version".
Standard brush-offs, and then finally the actual workaround... :)
Ginny Caughey [MVP] - 16 Oct 2007 15:11 GMT
Thanks for reporting back in case somebody else runs into this. It's good to
know it's fixed in the next version too.

Signature

Ginny Caughey
Device Application Development MVP

>> Is this a random example that causes a need to close and reopen the
>> connection (to avoid a GPF)?
[quoted text clipped - 9 lines]
> this", then "this is not a problem in the new (unreleased) version".
> Standard brush-offs, and then finally the actual workaround... :)
DRS - 10 Oct 2007 22:43 GMT
> Greetings,
>
[quoted text clipped - 17 lines]
>
> Thanks for your thoughts on this...

Answer: if your application is written in C++, go with SQLite (http://
www.sqlite.org/).  We did, and haven't looked back since.
Kay-Christian Wessel - 11 Oct 2007 21:36 GMT
We have experienced the same problems in our app which is in production. We
are uppdating the database very often by getting data from our SQL Server.
Our app is small and I'm sure we are disposing everything correctly.

Our CE device, which is running Windows CE 5.0 is crashing every 2-3 days.
Our workaround so far has been to reset the device from our program every
day.

The error we get is Out of Memory if we leave it running.

We hope to se a servicepack or a new version without these problems very
soon. Our customers are killing us.

Kay

> Greetings,
>
[quoted text clipped - 17 lines]
>
> Thanks for your thoughts on this...
Kay-Christian Wessel - 12 Oct 2007 22:03 GMT
After some testing we discovered that the device was working too hard and
the garbage collector probably had no processor power to do its work. After
releasing some processor power everything is working fine, and memory is
released as it should.

It seems that you can't put the same load on a CE device as you do on a PC.

Kay

> We have experienced the same problems in our app which is in production.
> We are uppdating the database very often by getting data from our SQL
[quoted text clipped - 33 lines]
>>
>> Thanks for your thoughts on this...
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.