SQL Server Forum / DB Engine / SQL Server CE / October 2007
Is it just me or are there BIG problems with SQLCE 3.0?
|
|
Thread rating:  |
An o' Neamus - 04 Oct 2007 07:52 GMT Greetings,
I wrote a very simple C# console application which was designed to give us a rough idea on the performance and reliability of SQLCE (the source code without metrics has been posted to this group recently). What I've found is that there are major performance and reliability issues with SQLCE 3.0 as opposed to SQLCE 2.0. Whereas SQLCE 2.0 seems to be fairly consistent in terms of it's memory usage and the amount of time taken to insert rows into the table regardless of the size of the table (to a limit, after about 300,000 records it too has major performance issues), SQLCE 3.0 seems to slow down noticably and continually use more memory even after a few thousand records are inserted and actually crashes (consistently) after 25592 records are inserted!
What gives? Based on this it would seem that SQLCE 2.0 is far preferable for an unattended device that needs an uptime of over 30 days whilst constantly (several times per minute) inserting into the database.
Thanks for your thoughts on this...
José Joye - 04 Oct 2007 20:50 GMT Looking at the code you posted previously, I have seen that your keep creating new strings in your Insert loop (500'000). This is really a memory killer habit. I do not know if this is the problem. However, try to look at this article for the best way to insert a larger amount of rows under CE
http://www.pocketpcdn.com/articles/articles.php?&atb.set(c_id)=74&atb.set(a_id)= 11003&atb.perform(details)=&
Hope this help - José
> Greetings, > [quoted text clipped - 17 lines] > > Thanks for your thoughts on this... joker - 04 Oct 2007 23:06 GMT A bad habit perhaps, but not the problem.
Even when creating a single string the problems still exists.
An o' Neamus - 05 Oct 2007 08:03 GMT On Oct 4, 8:50 pm, Jos? Joye <jose.joye__KILLTHESPAMS__SMAPSEHTLLI...@bluewin.ch> wrote:
> Looking at the code you posted previously, I have seen that your keep > creating new strings in your Insert loop (500'000). [quoted text clipped - 3 lines] > > http://www.pocketpcdn.com/articles/articles.php?&atb.set(c_id)=74&atb... Thanks. The actual application that this was a test for will run continuously and needs to stay up for at least a month. The test is a simulation of several single inserts that will be done per minute (of course I didn't actually want to wait a month, so I sped it up). So it's a valid test IMO because we will not be doing bulk inserts. The problem is not memory related, as (a) it fails in exactly the same place on multiple devices with varying memory sizes (and in some cases over 10Mb of program memory remaining), and (b) it doesn't fail on an older version of SQLCE (2.0). It's a bug in SQLCE 3.0, which we are not going to try and work around because if we find a serious bug with a simple program, then we are not going to take the risk of finding more bugs with more complex programs.
So to rephrase the question: As there is at least one show-stopper (for us) bug in SQLCE 3.0 that doesn't appear to be in SQLCE 2.0, as well as SQLCE 2.0 appearing to have vastly superior performance. Is there any reason why we would not want to be using SQLCE 2.0?
An o' Neamus - 05 Oct 2007 15:30 GMT Well, it would appear that this is definitely a SQLCE 3.0 problem. I've rewritten the same test in C++ using OLE DB, so the only constant is the choice of underlying database engine. It falls over (ie. 0xc0000005 exception) after inserting the exact same number of records! Using the SQLCE 2.0 engine with the C++ OLE DB program works as expected (ie. it works).
Also, taking the average time to insert a record (including flushes etc.), the SQLCE 2.0 engine appears to be between 3 and 10x faster than the SQLCE 3.0 engine (depending on the number of records in the table). The SQL CE 2.0 engine takes a constant amount of time regardless of the table size, whereas the SQL CE 3.0 engine gets progressively slower the more records there are in the table. The C++ OLE DB program appears to be between 100 and 200x faster than the .NET CF C# application when both are running against the SQLCE 2.0 engine (obviously the .NET CF application could be optimised to reduce the performance differential).
So it's a slam-dunk decision for us: OLE DB w/ C++ and SQLCE 2.0. Is there any advantage (to offset the poor performance and reliability) of using SQL CE 3.0 (assuming we don't need any of the new features)? Or is there a major disadvantage to using the SQL CE 2.0 engine? We do not have any need or intention to synchronise our database with a 'real' SQL server.
TIA
William Vaughn - 06 Oct 2007 18:19 GMT One trick you might try is to keep a connection to the database open. If you use the Open update Close technique, the database is compressed at the final close. Keeping a connection open postpones this operation.
 Signature ____________________________________ William (Bill) Vaughn Author, Mentor, Consultant, Dad, Grandpa Microsoft MVP INETA Speaker www.betav.com www.betav.com/blog/billva Please reply only to the newsgroup so that others can benefit. This posting is provided "AS IS" with no warranties, and confers no rights. __________________________________ Visit www.hitchhikerguides.net to get more information on my latest book: Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) -----------------------------------------------------------------------------------------------------------------------
> Greetings, > [quoted text clipped - 17 lines] > > Thanks for your thoughts on this... An o' Neamus - 08 Oct 2007 07:27 GMT > One trick you might try is to keep a connection to the database open. If you > use the Open update Close technique, the database is compressed at the final > close. Keeping a connection open postpones this operation. Thanks. The database was kept open for the entire test run. Another poster (in this or another thread I started on this subject), suggested closing the database occasionally to get around the problem we were seeing (ie. the database engine causing a GPF). As SQL CE 2.0 doesn't crash and is much faster, we've elected to use this version as opposed to SQL CE 3.0.
At this point, the only unanswered question we have is whether there are any reasons at all (given that performance and reliability are severely degraded) why we should even consider using SQL CE 3.0?
TIA
Erik Ejlskov - 08 Oct 2007 16:20 GMT Ordinary support for SQL Server 2000 (and with that SQL CE 2.0) ends in april 2008.
 Signature Erik Ejlskov Jensen - see my SQL Compact blog at http://erikej.blogspot.com
>> One trick you might try is to keep a connection to the database open. If >> you [quoted text clipped - 14 lines] > > TIA William Vaughn - 08 Oct 2007 20:00 GMT I encourage you to get up on MSDN Connect and start pounding on the SQL CE 3.0 team to help them repro the performance and crash cases you have experienced. I have passed your comments to the dev team directly but did not get much more than "It seems to run faster for us...". Since they have not shipped I think it would make sense for everyone to get these issues resolved.
hth
 Signature ____________________________________ William (Bill) Vaughn Author, Mentor, Consultant, Dad, Grandpa Microsoft MVP INETA Speaker www.betav.com www.betav.com/blog/billva Please reply only to the newsgroup so that others can benefit. This posting is provided "AS IS" with no warranties, and confers no rights. __________________________________ Visit www.hitchhikerguides.net to get more information on my latest book: Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) -----------------------------------------------------------------------------------------------------------------------
>> One trick you might try is to keep a connection to the database open. If >> you [quoted text clipped - 14 lines] > > TIA Ginny Caughey [MVP] - 08 Oct 2007 20:04 GMT Hi Bill,
It would also be important to have a repro example to test against 3.5 as well.
 Signature Ginny Caughey Device Application Development MVP
>I encourage you to get up on MSDN Connect and start pounding on the SQL CE >3.0 team to help them repro the performance and crash cases you have [quoted text clipped - 23 lines] >> >> TIA An o' Neamus - 09 Oct 2007 08:55 GMT > I encourage you to get up on MSDN Connect and start pounding on the SQL CE > 3.0 team to help them repro the performance and crash cases you have > experienced. I have passed your comments to the dev team directly but did > not get much more than "It seems to run faster for us...". Since they have > not shipped I think it would make sense for everyone to get these issues > resolved. Thanks. I've sent a message to the team through their blog as was requested on the forums.microsoft SQLCE forum. What is MSDN Connect, and how do I use this to 'pound' the SQL team? :)
I'm including a simplified example below for reference. Please note that the content of the example is not that important, this same test has been done with parameterised SQL, SqlCeRecordset and C++/OLE DB. It *always* fails at exactly the same point (ie. 25592 records), and never fails on SQLCE 2.0 (obviously the SqlCeRecordset test could not be done on SQLCE 2.0). As suggested by another poster, I've added a bit of code into my test which closes and reopens the database connection after 10000 inserts. This does solve the problem on SQLCE 3.0 (with all of the above access methods). However, I've read that opening and closing the DB connection is not a good idea. Will this introduce any additional problems, and if so, what are they?
TIA
Source code:
using System.Data; using System.Data.SqlServerCe;
namespace TestDB { class Program { static void Main(string[] args) { SqlCeConnection sqlConnection = new SqlCeConnection(); SqlCeCommand sqlCommand = sqlConnection.CreateCommand(); SqlCeResultSet sqlResult;
string strDatabase = "\\TestDB.sdf"; string strTable = "TestTable";
try { // Database connection string sqlConnection.ConnectionString = "Data Source=" + strDatabase;
// Database file doesn't exist? if (!System.IO.File.Exists( strDatabase )) {
Console.WriteLine("Creating database: " + strDatabase);
// Create SQL engine object SqlCeEngine sqlEngine = new SqlCeEngine( sqlConnection.ConnectionString);
// Use it to create database
sqlEngine.CreateDatabase(); }
Console.WriteLine("Opening database: " + strDatabase);
// Open database connection sqlConnection.Open();
// Display SQL version Console.WriteLine("SQL Version: " + sqlConnection.ServerVersion.ToString());
// Check if table exists sqlCommand.CommandText = "SELECT TABLE_NAME FROM " + "INFORMATION_SCHEMA.TABLES WHERE " + "TABLE_NAME = '" + strTable + "'"; sqlResult = sqlCommand.ExecuteResultSet( ResultSetOptions.Insensitive);
// Table doesn't exist? if (!sqlResult.Read()) { Console.WriteLine("Creating table: " + strTable);
// Create table sqlCommand.CommandText = "CREATE TABLE " + strTable + "(Sequence integer IDENTITY(1,1) NOT NULL PRIMARY KEY, " + "Timestamp datetime DEFAULT GETDATE()," + "Message nvarchar(80))"; sqlCommand.ExecuteNonQuery(); }
string strText;
for (int i = 1; i <= 500000; i++) { Console.Write("\rInserted " + i.ToString() + " records ");
strText = "Row Number: " + i.ToString();
sqlCommand.CommandText = "INSERT INTO " + strTable + " (Message) VALUES ('" + strText + "')"; sqlCommand.ExecuteNonQuery(); } } // SQL error catch(SqlCeException sqlex) { // Display all error messages Console.WriteLine( "ERROR:" ); foreach (SqlCeError sqlError in sqlex.Errors) { Console.WriteLine( sqlError ); } } // Other errors catch(Exception ex) { // Display error message Console.WriteLine( "ERROR:" ); Console.WriteLine( ex.Message ); } finally { // Close connection if (sqlConnection.State != ConnectionState.Closed) { Console.WriteLine("Closing database");
sqlConnection.Close(); } }
Console.ReadLine(); } } }
William Vaughn - 10 Oct 2007 20:32 GMT Connect is the feedback mechanism use to submit bugs to the dev teams. It's exposed in the help menus in Visual Studio.
 Signature ____________________________________ William (Bill) Vaughn Author, Mentor, Consultant, Dad, Grandpa Microsoft MVP INETA Speaker www.betav.com www.betav.com/blog/billva Please reply only to the newsgroup so that others can benefit. This posting is provided "AS IS" with no warranties, and confers no rights. __________________________________ Visit www.hitchhikerguides.net to get more information on my latest book: Hitchhiker's Guide to Visual Studio and SQL Server (7th Edition) and Hitchhiker's Guide to SQL Server 2005 Compact Edition (EBook) -----------------------------------------------------------------------------------------------------------------------
>> I encourage you to get up on MSDN Connect and start pounding on the SQL >> CE [quoted text clipped - 140 lines] > } > } An o' Neamus - 09 Oct 2007 18:12 GMT > One trick you might try is to keep a connection to the database open. If you > use the Open update Close technique, the database is compressed at the final > close. Keeping a connection open postpones this operation. Thanks. This doesn't appear to improve performance much (if at all). What does happen though, is that the amount of free memory on the device gradually decreases until it's about 240Kb. It doesn't completely run out, with 500Kb being freed before it gets a chance to completely run out. If you then do something else on the device that (in combination with the test program) causes the device memory to drop to about 140Kb, then (I guess it panics) and 1Mb gets freed. Rinse and repeat.
Same device, same program, one connection, closing the connection often, obviously slows it down, but keeps the available memory around 6Mb.
Is this a random example that causes a need to close and reopen the connection (to avoid a GPF)? Is the general rule to keep one connection open for the life of the program? Will adding complexity to the program and tables possibly change the environment such that the GPF might not happen (this example is obviously much less complex than the actual application's database needs)? Should we design the application so that it occasionally closes and reopens the database (bearing in mind it has to run indefinitely, and at least for 30 days)?
TIA
An o' Neamus - 16 Oct 2007 14:37 GMT > Is this a random example that causes a need to close and reopen the > connection (to avoid a GPF)? After some contact with the SQL CE team, it appears that this (GPF) problem is in fact a random example. The solution to the problem is to specify a value for the columns that have DEFAULT values (in this case the TimeStamp column), even if the value specified is the same as the DEFAULT (definitely a bug, but with a simple workaround).
It did take a fair amount of prodding to get the SQL CE team to provide a workaround. First, no response, next "we can't duplicate this", then "this is not a problem in the new (unreleased) version". Standard brush-offs, and then finally the actual workaround... :)
Ginny Caughey [MVP] - 16 Oct 2007 15:11 GMT Thanks for reporting back in case somebody else runs into this. It's good to know it's fixed in the next version too.
 Signature Ginny Caughey Device Application Development MVP
>> Is this a random example that causes a need to close and reopen the >> connection (to avoid a GPF)? [quoted text clipped - 9 lines] > this", then "this is not a problem in the new (unreleased) version". > Standard brush-offs, and then finally the actual workaround... :) DRS - 10 Oct 2007 22:43 GMT > Greetings, > [quoted text clipped - 17 lines] > > Thanks for your thoughts on this... Answer: if your application is written in C++, go with SQLite (http:// www.sqlite.org/). We did, and haven't looked back since.
Kay-Christian Wessel - 11 Oct 2007 21:36 GMT We have experienced the same problems in our app which is in production. We are uppdating the database very often by getting data from our SQL Server. Our app is small and I'm sure we are disposing everything correctly.
Our CE device, which is running Windows CE 5.0 is crashing every 2-3 days. Our workaround so far has been to reset the device from our program every day.
The error we get is Out of Memory if we leave it running.
We hope to se a servicepack or a new version without these problems very soon. Our customers are killing us.
Kay
> Greetings, > [quoted text clipped - 17 lines] > > Thanks for your thoughts on this... Kay-Christian Wessel - 12 Oct 2007 22:03 GMT After some testing we discovered that the device was working too hard and the garbage collector probably had no processor power to do its work. After releasing some processor power everything is working fine, and memory is released as it should.
It seems that you can't put the same load on a CE device as you do on a PC.
Kay
> We have experienced the same problems in our app which is in production. > We are uppdating the database very often by getting data from our SQL [quoted text clipped - 33 lines] >> >> Thanks for your thoughts on this...
|
|
|