Posted
13 days
ago
by
Rodrigo B. de Oliveira
Db4o for .NET 3.5 ships with an optimizing LINQ provider. This provider is able to optimize the execution of a query by analyzing the LINQ expression tree emitted by the C# (or VB.NET) compiler to produce an equivalent SODA query which can be
... [More]
executed by db4o at full speed.
The Compact Framework 3.5 however includes no support for expression trees. No expression trees, no optimizing LINQ provider. A saddening state of affairs for Compact Framework db4o users.
As it turns out, expression trees are provided by the System.Linq.Expressions namespace which is not available in the standard Compact Framework 3.5 distribution.
Mono to the rescue
The Mono project provides a free software implementation of the .net framework including LINQ and the System.Linq.Expressions namespace.
Hypothesis: will the Compact Framework compiler emit expression trees if it sees a reference to a compatible assembly that provides a System.Linq.Expressions namespace?
Yes it will. LINQ for Compact Framework seemed to be possible.
There was a remaining issue. When the db4o LINQ provider cannot optimize a query down to SODA it can simply execute it as a LINQ for Objects query. This is accomplished by invoking the Compile method on the expression tree. The returned delegate can be used to evaluate the expression whenever needed. But the Compile method relies on System.Reflection.Emit which is again not supported by the Compact Framework. Hmpf.
The final piece of the puzzle appeared when Mainsoft contributed an expression tree interpreter to Mono. Expression tree evaluation that didn't rely on System.Reflection.Emit. Yummy.
So thanks to the hard and loving work of the Mono and Mainsoft guys we are able to ship to you the first optimizing LINQ provider for the Compact Framework.
A nice hack indeed. [Less]
Posted
25 days
ago
by
Patrick Roemer
Finally we are starting to harvest the fruits of our work on pluggable type handlers. As a first step, please welcome .NET generic collections to the exclusive club of types natively supported by db4o in the upcoming 7.6 build.
In principle
... [More]
, .NET generic collections could already be persisted in previous versions, but there was no dedicated handling in place for them. db4o would not recognize them as collections and would simply handle them as any other object - inspect their internal layout and persist the field values found. This approach came with some pitfalls. Most notably, db4o would not recognize the collection's internal state objects as parts of a single, atomic unit - in C/S mode, concurrent updates from multiple clients against the same collections could have resulted in inconsistent internal collection state in the worst case. Furthermore, issues could arise with specific collection implementations: Dictionaries, for example, would persist hash codes alongside keys/values and could stumble when attempting to reload this data in a different runtime version (as GetHashCode() implementations may change yielding different hash values). Plus, last but not least, dedicated collection handling opens the path for performance optimizations.
With the current type handler infrastructure in place, providing custom handling for .NET generic collections as its first "real world" application went pretty smooth - actually the bulk of the effort was spent on the implementation of combinatorial regression testing of permutations of collection, field and element types - take a look at GenericCollectionTypeHandlerTestSuite, if you're interested. As of now, dedicated type handler support is available for the following collection flavors:
List<T>LinkedList<T>Stack<T>Queue<T>Dictionary<K,V>SortedList<K,V>SortedDictionary<K,V>
These collections now should blend in seamlessly with the behavior of non-generic collections. More to come - but if you're missing a dedicated type handler for a specific flavor, of course you're also invited to contribute. ;) [Less]
Posted
about 1 month
ago
by
Carl Rosenberger
here is a short info to resolve a race condition between our newsletter sendout and the very latest production build:
We got informed about a serious (blocker) bug with a deadlock in committed callbacks after our newsletter had already been
... [More]
released to marketing: COR-1377
This bug received the highest priority and it has has been fixed immediately.
A new production build with the build number 7.4.58.11547 is online in the meanwhile. If you downloaded 7.4 right after the newsletter and if you use committed callbacks or UUIDs, we recommend to check the build number you are using (call Db4o#version() / Db4oFactory#Version() ) and to download the very latest build once again if it is lower than 7.4.58.
To our knowledge this bug has been present in all 7.2 builds. [Less]
Posted
about 1 month
ago
by
Carl Rosenberger
Two months ago we invited the db4o community to take part in the first db4o performance contest. Now the contest has been completed. Here are the results.
Three entries are eligible for prizes:
June 15Andrew Zhang submitted a patch that
... [More]
improves the speed to sort queries.
June 19Andrew Zhang submitted a UTF-8 string encoder.
July 17Erik Putrycz submitted a patch to reuse query results in subsequent similar queries.
Andrew and Erik wrote own Poleposition circuits for their patches to be able to measure performance improvements. They have been incorporated into the db4opolepos project in our SVN as circuits "Hockenheim" and "Hungaroring".
The three patched jars that Andrew and Eric supplied have been renamed to db4o-7.2-sortedquery.jar, db4o-7.2-utf8.jar, and db4o-7.4-cachedqueries.jar to make it easy to distinguish the results. These jars have also been checked into the db4opolepos project. Anyone can easily reproduce the performance race by running org.polpos.AllRacesRunner from this project.
The results from running this benchmark on our hardware are attached to this posting as PerformanceContest.pdf.
Our impression from reviewing the results and the patches:
- The effect of the tuned quicksort algorithm on query sorting is absolutely amazing. Improvements are indeed in the range of 20x to 300x as you can see in Hockenheim#query_ascending and #query_descending on pages 52 and 53 of the attached Poleposition results. The patch is very straightforward and very nice, so it can immediately go into production code as is. A perfect submission.
- The UTF-8 string encoder reduces the required slot length for stored strings. Shorter slots result in faster writes as you can see in Nurburgring#write on page 24 of the attached Poleposition run. A performance improvement of about 20% is measurable. Incorporation of the patch code into the db4o core would require some more work: Probably we would add UTF-8 as a third configurable string encoding. Possibly we could make it the default. In any case we would have to continue to support the existing Unicode string encoding. The submission shows very nicely how easy a UTF-8 string encoder is. We should make one available soon.
- The idea to cache query results is great. A working implementation is certainly an order of magnitude more difficult to write than the other two submitted patches. Somehow the Hungaroring benchmark shows a slowdown instead of an improvement. After Erik's comments that our Trunk code got slower for concurrent queries, we have also compared running the patched Jar against Trunk and Trunk still turns out to be faster. Erik's submission also does not exactly follow the rules of the contest: The patch is not applied to the 7.2 workspace, so a direct comparison run to other submissions is not completely fair. In any case we really liked the idea and we have to give credit for trying.
So how do we judge the three submissions and who gets first, second and third prize?
Andrew's submissions both are measurable and they are both in compliance with the rules that we have set up. Erik's submission is a great idea, but it is not 100% complete and we have not yet seen the improvement clearly documented in a Poleposition run.
The core team reviewers suggest to award prizes as follows:1st USD 3000 Andrew Zhang2nd USD 2000 Andrew Zhang3rd USD 1000 Erik Putrycz
If anyone in our community has a different opinion, please post it to our performance forum.
Andrew and Erik: Thank you very much for your submissions. You have deserved the prizes!
For all other developers that tried to improve performance but did not complete a submission on time: Thank you for trying!If you have suggestions how to improve the Performance Contest, please tell us about them. Maybe it would make sense to let the contest run for a longer period of time than 2 months? What do you think?
We are very happy to have received usable patches in this first db4o performance contest. Hopefully we will get even more contributions the next time.
It is great to work with such a nice community! [Less]
Posted
about 1 month
ago
by
Carl Rosenberger
You were playing with the idea to contribute to db4o? So far you did not find a nice, easy and short task to take on? Here is your chance: Write a TypeHandler for db4o and get your code included in the db4o core distribution tomorrow.
db4o comes
... [More]
with a big promise: Just call ObjectContainer#store() and any object and all attached objects will be automagically stored. The challenge to do this is high: Every possible class out there needs to be "understood" by db4o and an efficient byte array representation needs to be created on storage media. Indexing of "parts" of objects and querying is part of the task. With evolving languages and an increasing number of class constructs that need to be supported we decided to make it easier to write the code for marshalling and unmarshalling: Enter TypeHandlers.
TypeHandlers define how objects of a specific class are stored, loaded and queried by db4o.
By default db4o stores and restores all non-transient fields of an object. By writing TypeHandlers it is possible to alter the behaviour. A TypeHandler could decide to store only some members of a class or it could decide to store a completely different and more efficient representation.
The possible benefits of using a Typehandler:- Storing and restoring objects can be faster.- The database file can be smaller.- Objects that consist of sub-objects (Example: collections) can be guarded against inconsistencies in concurrent multi-user environments by preventing "the object" from being split, but using one single contiguous slot as the marshalled representation. ( We will soon also demonstrate a stateful collection implementation over multiple pages, also powered by a Typehandler.)- Fields that are intended to be transient do not need to be stored.- Typehandlers can alter behaviour even if the sources of the persistent classes are not available or can not be altered.- Querying support.
TypeHandlers are expected to do the complete marshalling of an object by writing/reading directly to/from the marshalling buffer.
About a year ago the TypeHandler4 interface consisted of more than 40 methods. Today only 5 methods remain. To make sure that the TypeHandler interface is ready for heavy duty use, we have already used it ourselves to implement some collection typehandlers. See:com.db4o.typehandlers.CollectionTypeHandler (Java)com.db4o.typehandlers.MapTypeHandler (Java)Db4objects.Db4o.Typehandlers.ListTypeHandler (.NET)Db4objects.Db4o.Typehandlers.MapTypeHandler (.NET)
In the latest development build (7.5.x.y) these TypeHandlers are already turned on to do real work storing, loading and querying some selected collections. To see how internal TypeHandlers are configured, take a look at:com.db4o.internal.TypeHandlerConfigurationJDK_1_2 (Java)Db4objects.Db4o.Internal.TypeHandlerConfigurationDotNet (.NET)
You will notice that many TypeHandlers are still missing if the full featureset of JDK and .NET collections is to be supported. Here is where we would like to ask you for your help:Could you write a TypeHandler for us for a collection class that you think we need to support?
Adapting and registering one of the already existing typehandlers for your favourite collection will probably take you less than an hour of work. You can simply follow the pattern that is already there.
However writing the TypeHandler code is not the complete story. Since we want to be absolutely sure that a new TypeHandler works correctly, the bigger part of the implementation work is required for testing. We want to make sure that objects stored with an older version of db4o can be read and stored even after your new TypeHandler is installed.
For this purpose we developed the "Db4oMigrationTestSuite" test.On Java you can run this class directly. You will find it in our SVN in the db4ojdk1.2 project. ( If you have not worked with the db4o sources before, here is a guide to get started. )
If you develop for .NET you would check out the db4o.net project from our SVN. You would typically work with the Db4o-2008.sln solution in the top level folder. To run the tests, the Db4objects.Db4o.Tests-2008 project would be your startup project. To save time by not running all the tests every time you can edit Db4objects.Db4o.Tests.AllTests in the Db4objects.Db4o.Tests-2008 project: If you want to run migration tests only, you can comment out all the entries in the #TestCases() method and add the following instead:typeof(Db4objects.Db4o.Tests.Common.Migration.Db4oNETMigrationTestSuite)
Both for Java and .NET: If you want the Db4oMigrationTestSuite to truely run against all relevant old db4o engines, you should check out the db4o.archives project from our SVN also.
Once you have testing set up and running, you are ready to go to write your own test for updating your favourite collection.
For an example how an update test could look like, please see:com.db4o.db4ounit.common.handlers.ArrayListUpdateTestCase (Java)Db4objects.Db4o.Tests.CLI2.Handlers.GenericListVersionUpdateTestCase (.NET)
Only when you have the test in place it makes start to implement your own TypeHandler as a more efficient method to store the collection. Test first, remember?
Should you have any problems to get started, please ask for help in the db4o product developer forum. It is a good idea to announce any work that you are about to do in the forum in any case. That will ensure that the work for a specific collection is only done once.
Now that you are all set to start:Good luck for writing your TypeHandler!
All the new Typehandlers we have built so far are now available in the TRUNK of our SVN and with our continous build.
They help to make storage of collections more efficient:- Less state is pushed around than with our previous translator approach.- Maps no longer store an internal Entry object for each entry.- All .NET collections, including all .NET generic collections will soon be handled efficiently with Typehandlers. Current issues with .NET generic collections in concurrent multi-user setups will soon be history.- Old database files can be read with new Typehandlers installed. Future marshalling format changes are easily possible by writing and installing yet further new Typehandlers.- We have a very nice hook into collection treatment that we will be able to use for much faster collections.
Enjoy! [Less]
embedded c# java database object persistence database_server replication server development dotnet dbms db4o
|
edit tags
