Mats Helander Weblog

Ok, I overreacted a bit in my previous post. I have been thinking about what it could be I really reacted about.

For a long time, Microsoft experienced a bit of a tarnished reputation among open source enthusiasts. Among some, they still do, but without being a MS historian, somewhere around the beginning of the .NET era I think that many of us got the message that MS was hoping to change the perception of the company as something of a “bully” - and that this to some extent included a less “blanket draconian” stance towards open source. Ok, perhaps Linux wasn´t exactly great for MS, but that wouldn´t necessarily have to mean that _all_ open source initiatives were to be depicted as disruptive anymore.

I was very, very happy with this development, because as I met more and more people from MS in my work and on conferances, they invariably turned out to be exceptionally great people! Really, smart, talanted and (especially given MS’s bully reputation) strikingly humble and cool.

When I started to work on my own open source initiative, my experience was that MS had been so successful in its persistent efforts to improve its reputation that when I met “softies” and my open source stuff came up we could share a hearty laugh about how perhaps “them might´ve been fightin´ words” in the bad old days and how nice it was that those dark ages were over.

So I think my overreaction was a kind of “Nooooooo!” moment where I suddenly extrapolated to MS taking a new direction again, back towards a more hostile attitude towards all things open source. And then I focussed that fear on the individual behind the quote.

So, Mr Ferreira, while I hope that my humble blog flies well below your radar, I want to apologise for completely overreacting to your statements! Sorry!

Update: My tone in this post was unecessarily harsh. Please read my Apology to Mr Ferreira. 

According to Microsoft, apparently “there’s no such thing as free software” and “Nobody develops software for charity”. The quotes are from Mr Paulo Ferreira, the platform strategy manager at Microsoft South Africa.

http://mybroadband.co.za/news/Software/3258.html

It’s hard to know where to begin. I feel tempted to go on a long tirade and explain just how motivated I feel to brush up on my Java after reading crap like that.

It is extremely offensive to maintain the position that lots of people apparently just don’t exist. It is as if we are standing in front of Mr Ferreira, waving our arms and going “hey, you missed us, here we are” and he goes “Nah. They don’t exist.” Alternatively, “Well, they might exist. But they’re nobodies.”

That’s why it amuses me that according to this genius MS rep, Microsoft doesn’t exist, either.

The Bill and Melissa Gates foundation is a charity organization founded by Bill Gates and his wife. They spend quite heavily on charity, to say the least. They do so using the money that they have earned from Microsoft. That is, from software development.

In other words, possibly more so than any other company on the planet Microsoft develops software for charity. Which, according to themselves, apparently makes them non-existing or possibly “nobodies”.

Is Bill aware that one of his reps just called his wife a “nobody” ?

Michael Tiberg, the man behind the amazing OreDev conference, has just let me know that the presentations from the latest conference are now publicly available online!, including my presentation Domain Model Management - AOP in Practice.

As the name suggests, the topic of the presentation is similar to that of my InfoQ article Aspects of Domain Model Management.

To the audience members who contributed with questions: My deepest apologies for the fact that I still just can’t get that most basic of presentations skill right: Always repeat the questions from the audience - I can hear you but the rest of the audience can’t! One of these days, I will learn…

I just read the following on Krzysztof Cwalina’s blog:

For example, the C#’s foreach operator already uses duck typing. This might be surprising to some, but to support foreach in C# you don’t need to implement IEnumerable! All you have to do is:

Provide a public method GetEnumerator that takes no parameters and returns a type that has two members: a) a method MoveMext that takes no parameters and return a Boolean, and b) a property Current with a getter that returns an Object.

For example, the following type supports foreach:

class Foo {
    public Bar GetEnumerator() { return new Bar(); }

    public struct Bar {
        public bool MoveNext() {
            return false;
        }
        public object Current {
            get { return null; }
        }
    }
}

// the following complies just fine:
Foo f = new Foo();
foreach (object o in f) {
    Console.WriteLine(“Hi!”);
}

I just couldn’t believe what I was reading.

What this means is that foreach will not work with a class that explicitly implements IEnumerable – the compiler will complain that your class doesn’t implement public methods with “magic” names.

In other words, while the example above by Krzysztof’s blog will compile, the following (that I would much rather see compiling) won’t:

class Program
{
    static void Main(string[] args)
    {
        // the following does not complile!
        // all because of foreach duck typing
        // rather than properly using interfaces…

        YearCollection f = new YearCollection();

        foreach (object o in f)
        {
            Console.WriteLine(”Hi!”);
        }

        Console.ReadLine();
    }
}

class YearCollection
{
    public Year GetEnumerator() { return new Year(); }

    public struct Year : IEnumerator
    {
        public object Current
        {
            get { return new DateTime(DateTime.Today.Year, 1, 1); }
        }

        private int cursor;

        bool IEnumerator.MoveNext()
        {
            if (cursor >= DateTime.Today.Year)
                return false;

            cursor++;
            return true;
        }

        object IEnumerator.Current
        {
            get { return new DateTime(cursor, 1, 1); }
        }

        void IEnumerator.Reset()
        {
            cursor = 0;
        }
    }
}

I don’t even know how to begin explaining how wrong this is. I’m not automatically against duck typing when it makes sense, but to just ignore the intended interfaces in a statically typed language like C#…well, I would love to hear the rationale behind that design decision.

Fortunately there’s a workaround: the code above will work if I change the return type of the GetEnumerator() method to IEnumerator.

Normally you might expect the consumer to try to dynamically cast the return type of the provider to the required interface. Not so with foreach - here it is suddenly the job of the provider to cast the return values to…well, I was going to say the types expected by the consumer, but I guess it is in fact a case of “any type matching the duck type (signature) expectations of the consumer”. I would consider that a fairly…unusual design decision.

So the question I’m left with is…why have a type system if you’re not going to use it?

This post is a reply to a comment by Krzysztof Cwalina in Ayende’s blog

Hi Krzysztof,

I will try to get back with a longer answer later but for now let’s just say I agree with Jeffrey Richter in his concerns and reservations regarding the conclusions you and his other co-authors draw in the particular section covering this discussion, in chapter 4 of the Framework Design Guidelines:

http://www.theserverside.net/tt/articles/content/FrameworkDesign/Chapter4.pdf

I predict that when I get back to trying to explain what I think is problematic with your approach, it won’t be nearly as neatly summed up as Jeffrey’s remarks, but I have to say they seem to me, in all their succinctness, to completely contradict (and even, perhaps, invalidate) the rest of the advise given in that chapter?

I couldn’t actually see anything that looked like an appropriate response to the points raised by Jeffrey in that chapter, so, well, I guess to begin with I would like to know if you think 1) All of Jeffery’s remarks have actually been properly addressed in the guidelines 2.a) If so, where? 2.b) If not, why not?

If you can clarify this for me, my answer can hopefully become shorter and more to the point.

In particular, I am thinking about:

“My recommendation to people is this: Define an interface first and then define an abstract base class that implements the interface. Use the interface to communicate to the object and let end-user developers decide for themselves whether they can define their own type based on your abstract base class (for convenience) or define their own type based on whatever base class they desire and implement the interface (for flexibility)”
/Jeffrey Richter, Framework Design Guidelines, Chapter 4 section 4.3

That advice sounds just like music to my ears. The only section in the main text that seemed to deal with Jeffrey’s suggestion was the following part, attempting to describe why you shouldn’t create a new interface for the new functionality - what many (incuding me and, unless I’m misreading him, Jeffrey) might go so far as to call the “proper” approach:

“This option unfortunately does not perform well in usability studies. The fact that some streams [The example abstract base class, my comment] can now support the new operation is not immediately visible to the users of StreamReader APIs. Also, some developers have difficulties understanding and using dynamic casts.”

But surely this can’t be the actual reason, because I just can’t imagine that you would argue in a book about framework design guidelines that what (as far as I’m aware?) is generally considered a solid and time honored OO principle (and which is described very well in the direct advice to use this approach by Jeffrey) should be discarded on account of it suddenly “not performing well in usability studies.”..? I really do hesistate to ask this but….were those usability studies by any chance performed during the time MS said it was having a tough time “teaching all those old VB6 programmers OO”?

To make this clear - the approach you dismiss because it didn’t fare well in your usability studies appropriately addresses the scenario that you want to solve by using abstract base classes instead of interfaces, but you prefer base classes because…

if (reader.BaseStream.CanTimeout)
{
reader.BaseStream.ReadTimeout = 100;
}

is better than

ITimeoutEnabledStream stream = reader.BaseStream as ITimeoutEnabledStream;
if (stream != null)
{
stream.ReadTimeout = 100;
}

…because CanTimeout will show up in IntelliSense…? Is this correct?

The framework guidelines chapter doesn’t seem to mention in a clear way when promotning the abstract base class alternative version of the code (?) that developers missing to note the CanTimeout member showing up in the IntelliSense but do happen to notice the new ReadTimeout property and happily go on to access it may get NotImplementedExceptions thrown in their faces at runtime - whereas it is less likely, I would argue, that the developer using the “as” keyword to cast would forget to do the null check. Phil Haack, however, does address this concern in his post on the topic with the words “hey, thems the breaks. Remember, no perfect design.”

Phil also talks in his post about how we must understand that different constraints can apply, and that in this case “no breaking changes” would be a stronger requirement than the improved flexibility and substitutability offered by the interface based approach. But the interfaced based approach (adding a new interface for the new functionality) doesn’t bring any breaking changes - it just requires that the developers understands enough of OO to be reasonably comfortable with dynamic casting. (And, one might add, if they are not, perhaps they shouldn’t be expected and trusted to remember to look for and find all the CanDoXXX methods before calling XXX methods, risking NotImplementedExceptions showing up in runtime.)

So, I guess what this boils down to is:

Are we to understand that you decided to sacrifice the flexibility that Jeffery refers to in his quote above for the “constraint” of “not being able to use the standard OO mechanisms” because “according to your usability studies your customers didn’t understand standard OO mechanisms” ?

My girlfriend Theza has started a new photo blog where she publishes pictures that she takes on her walks in our new home town, Amsterdam.

Visit her blog here: http://1054amsterdam.blogspot.com/

Please drop a comment in her comment section and say how you like the pictures!

NPersist has just been updated with a few new features that motivate stepping up the version number from v1.0.9 to v1.0.10. Inside NPersist has also been updated to cover the new features (as well as with a new section detailing the workings of an already existing aspect of the Inverse Manager - how it deals with lazy synchronization of inverse properties).

The new features in NPersist v1.0.10 are:

Read and Write Consistency Enforcement

The IContext interface has two new properties called ReadConsistency and WriteConsistency, each taking a ConsistencyMode enum which can be Optimistic or Pessimistic (or Default, which translates to Optimistic).

Optimistic Read Consistency (the default) is what NPersist has been doing so far – taking an optimistic approach to the consistency of data read into memory, not throwing exceptions to the left and right because objects are loaded outside of or in different transactions.

If you set ReadConsistency to Pessimistic, however, NPersist will enforce that objects that reference each other in a graph are not loaded during different transactions. The main use case this is supposed to address is when you want Lazy Loading rather than “Lucky Loading”, which is in all fairness what NPersist has been giving you to date.

The problem with Lazy Loading is that Dinesh was right when he concluded it often represents something of a Wild West attitude to data consistency.

Sometimes all you care about is data consistency in the database. As long as the proper consistency rules and constraints are in place to ensure that inconsistent data is not entered into the database, you’re good. In such a case, you’d probably go with Optimistic Read Consistency.

But other times it is important to ensure that the data in memory is consistent and accurate – perhaps because the in memory version of the data is used for some business process with real world side effects, such as sending out bills to customers.

In such cases, you would normally make sure that you read up all the data that you are going to need for the process to complete within a single transaction. Assuming the appropriate isolation levels are used, this ensures that we get a version of the data that nobody could have modified while we read it.

This doesn’t actually preclude lazy loading, however – it only means that the lazy loading will also have to take place within the transaction. Lazy Loading can help you avoid having to figure out exactly what data you’re going to need in advance (especially when that depends on sometimes complicated evaluations of some of the other data that should be fetched first) and having to write explicit code for fetching the conditional data (just accessing the properties will do). However, if you want Pessimistic Read Consistency, you’re going to want to be sure that the Lazy Loading doesn’t take place outside of the transaction.

So that’s what NPersist does when you set context.ReadConsistency = ConsistencyMode.Pessimistic – it throws a ReadConsistencyException if you try to Lazy Load outside of the right transaction.

In summation, a ReadConsistencyException will be thrown under the following conditions when ReadConsistency is set to Pessimistic.

• If an object is loaded or created outside of a transaction.
• If a property is lazy loaded outside a transaction or in another transaction than the object that the property belongs to was loaded or created in.
• If a property is loaded with a reference to an object that was loaded or created in another transaction than the object that the property belongs to was loaded or created in.

When it comes to Write Consistency, again we see that Optimistic Write Consistency is what NPersist has been giving you so far (and, again, it will remain the default behavior so that by updating to the new version of NPersist you will not immediately get ten million exceptions thrown in your face). Simply put, with Optimistic Write Consistency objects don’t have to be loaded and saved in the same transaction – instead we rely on Optimistic Concurrency to maintain write consistency (which is turned on by default in NPersist).

Pessimistic Write Consistency means that you won’t be using Optimistic Concurrency – instead NPersist will throw a WriteConsistencyException if you try to persist an object outside of the transaction it was loaded or created. In summation, a WriteConsistencyException will be thrown under the following conditions when WriteConsistency is set to Pessimistic.

• If an object is loaded or created outside of a transaction.
• If a property is written to outside of the transaction that the object that the property belongs to was loaded or created in.
• If a property is written to with a reference to an object that was loaded or created outside the transaction that the object that the property belongs to was loaded or created in.
• An object is inserted, updated or removed from the database outside the transaction that the object was loaded or created in.

Read and Write Consistency enforcement gives NPersist potential to address a wider range of use cases where higher levels of consistency should be enforced both for in memory and in database data.

Note that you have been able to address both these scenarios before (pessimistic read and write consistency) by encapsulating your operations correctly with transactions – it is just that NPersist would not help you out with enforcing this consistency before by throwing exceptions if you made a mistake.

Thus, when you have existing code where Pessimistic Consistency modes should be switched on, you probably know it already and have the appropriate transactions in place – turning Pessimistic Read/Write Consistency on will then help you enforce that consistency.

Eager List Count Loading

When NPersist loads an object with values from the database, it begins by loading the values for its non-list properties, whereas list properties default to being loaded lazily on first access. However, as of v1.0.10, NPersist will now default to loading the Count values for those list properties directly (“eagerly”) along with the non-list values when the object is first loaded.

NPersist does so using sub-selects in the SQL, so that loading a Customer with ID 42, the SQL might look something like this:

Select ID, FirstName, LastName, Email, (Select Count(*) From Orders Where Orders.CustomerID = Customers.ID) As OrdersCount From Customers Where ID = 42

This means that if you access just the Count property of the list object in one of your list properties that hasn’t been loaded with items yet, no lazy loading of the list has to be triggered. Before, NPersist would load the list with items from the database in order to be able to answer the call to the Count property whereas now this can be answered using the eagerly fetched count value.

Since your immediate reaction to this is probably that it is a horrible idea and you ask yourself how you can switch it off, the answer is: In the ListCountLoadBehavior property on the PersistenceManager (set it to Lazy: context.PersistenceManager.ListCountLoadBehavior = LoadBehavior.Lazy) or, if you prefer doing it in the xml mapping file, use the new “count” attribute in the domain element (. If you think the idea has merit but only for a few of your classes or properties, the “count” attribute is present on class and property level as well in your xml mapping file. Please refer to Inside NPersist for further explanation on how to use these new attributes).

This change of default behavior is why I felt the version of NPersist should be stepped up from v1.0.9 to v1.0.10 (we have otherwise had a decidedly, ahem, “conservative” (lazy) approach to stepping up version numbers) - to draw attention to it so you don’t get this changed behavior without noticing it in case you really don’t want it.

But before you rush to turn it off, allow me a minute to explain why I think this new behavior is useful and why making it the new default behavior is a good idea.

First off, it can obviously save you a fairly expensive lazy loading of list items in lists whenever you have code that looks something like this:

if (myCustomer.Orders.Count < 1)
return;

If that would be the only dividend, though, I would have agreed that eager list count loading should be off by default and turned on explicitly only where you have situations like that one.

The reason, rather, that defaulting to eagerly loading the list counts is that the third new NPersist feature - that I’m going to describe now - can make use of it to reduce the number of list properties being triggered to lazy load even further.

Inverse Property Resolution

Say that you load a number of customers from the database. Then you go on to load a number of orders from the database. Ten of the orders you load up happen to belong to the same customer, who happens to be loaded into memory by the previous query. Furthermore, those ten orders happen to be all the orders belonging to that customer. In other words, NPersist should be able to use these orders to fill the Orders property of the customer they belong to, so that when that list property is accessed it doesn’t have to be lazy loaded.

But the problem is – how could NPersist know that it has all the orders it needs to fill the Orders property of the customer? Well, if the count property for that Orders property had happened to have been eagerly loaded, that would allow NPersist to know that whenever it has found ten orders belonging to that customer, it could mark the property as loaded and fill it with the orders.

The orders do not even have to be loaded as part of the same query – the ten orders can come from many different queries spread out over time. But hold on, the paranoid developer may object at this point: What if the number of orders belonging to the customer changes in the database between these queries – couldn’t that give funny results?

Why, yes it very well could – glad you pointed it out, Mr Paranoid Developer! That’s why this new feature will also respect the new Pessimistic ReadConcurrency setting, should you choose to activate it.

Because this effect is really no worse than all the other inconsistencies of just the same kind that can happen to in memory graphs of the data if you allow the graphs to be partially loaded outside of (or in different) transactions. This is a good illustration of why Dinesh is right to feel skeptical about lazy loading.

However, in many situations it really isn’t catastrophic so long as you don’t end up with corrupt data in the database, and for all those situations you’re really not much more exposed to “weirdness” using this new feature than the standard lazy loading outside transactions. And when any such weirdness is highly undesirable, just set ReadConsistency to Pessimistic.

The module in NPersist responsible for knowing and filling list properties based on their eagerly loaded count values is, of course, the Inverse Manager, since it exploits the inverse property meta data information in order to do this.

Loading an order, the Inverse Manager uses the knowledge that the customer.Orders property is the inverse of the order.Customer property that is being filled with a reference to the customer when the order object is loaded to know that it can add a back reference to the order to the customer’s list property (making that list property partially loaded until it reaches the number of items matching the eagerly loaded Count value).

The opportunity for the inverse manager to intelligently resolve inverse references like this when objects are being loaded is new to NPersist – until now, the Inverse Manager has only been involved in synchronizing inverse properties when you write to them, never really getting involved when you loaded objects from the database (except for the Lazy Synchronization feature, described in the new Lazy Inverse Property Relationship Synchronization section in Inside NPersist).

The new Inverse Property Resolution feature of the Inverse Manager has the potential to significantly reduce the number of lists that cause a lazy loading operation being triggered on the first access to them. It also resolves reference properties (not just lists) meaning for example that One-One relationships will also be resolved by this new responsibility of the Inverse Manager - and resolving those relationships doesn’t even rely on eager list count loading being turned on.

But being able to resolve inverse relationships involving list properties is of course the main reason that eager list count loading is turned on by default. It is the feature that enables the Inverse Manager to perform Inverse Property Resolution on list properties.

The longtime reader of this blog may recall that the genius behind all this is of course in accordance with ancient tradition Roger Alsing (then Johansson) who came up with all this in a response to my O/R Mapping Challenge. As usual, I was merely the code monkey

However, the longtime reader with uncanny eye for detail may also remember that Roger’s suggestions (which have now been implemented) don’t actually meet the requirements of my challenge – the sub-selects in the SQL are strictly not allowed by my own stupid rules

However, they do allow for a far more powerful Inverse Property Resolution than a feature which would pass my challenge and which I am currently pondering an implementation for:

In the special case that all objects of a type are loaded (the conditions specified in my challenge) NPersist could add the information to an internal structure that for all inverse list properties referencing objects of this type, the Inverse Manager could go ahead and do its job (it wouldn’t need the eager counts for this since this is a special situation that would allow it to know that all objects for the lists would be there).

I’m not sure if I should add it, since it does feel like a kind of contrived optimization hack, perhaps more put in place to meet my own challenge than provide benefits for the users of NPersist…but on the other hand it would allow for a very efficient handling of that special case and in my opinion it isn’t even really a corner case scenario (some CMS load all active data for articles and such into memory in the front ends at application startup). So I may go ahead and implement it after all, finally making NPersist join the group of mappers able to pass my own challenge

Nonetheless, the current new set of features should be widely useful and help improve the overall performance and consistency of NPersist. And NPersist almost passes my challenge now!

Let me know what you think!

Yes indeed! After more than a year of downtime, Roger Alsing - formerly Johansson - is back in the blogosphere!

His new blog is hosted at:

http://www.rogeralsing.com

Since Roger, in contrast to me, is equipped with so called “sensibilities” he can actually make a web page look nice. Thus the very first thing I did when his new blog was up (before posting this entry with the link) was to steal what I could from his page design. So, how do you like my new look?

Furthermore, since we have decided that LINQ to SQL just isn’t a valid replacement for NPersist in many scenarios we have decided to continue development. So you can expect some more Puzzle news ahead…but a first piece of news is that I have done some new performance optimizations that can potentially double the speed of NPersist in many standard operations. (Primarily improvments in how meta data that is the result of calculations in the meta data DOM is better cached). Oh, and the events now finally work as advertised   [Patrik, I do believe the DateTime null value bug should be fixed as well! ]

I’m throwing my old CV out the window. To future potential employers I will simply forward the following link:

http://www.c64gg.com/P15.html

Scroll down to the second “Pong”.

I don’t remember the year I wrote it, but I could help complete the stats on the web page by recalling that the “distribution media” was an old cassette tape. I have no idea who might have made a copy to someone who made a copy to someone so it finally ended up on this web site, but I can tell you I was pretty surprised when I found it.

More noteworthy trivia about the game: It was written in 100% machine code on my commodore 64 and the somewhat cryptic message “numbersprgfx by bobo” in the screenshot from the menu screen above means that the sprite graphics for the numbers, seen in the “in game” screenshot below, were made by my brother (his nickname is Bobo).

Yes, Frans, those are colorbars in the pong logo. Yes they bounced up and down in a sinus pattern. No you simply can’t imagine how great the “starlit sky” background, sadly not featured in the available screenshots, looked.

I am very proud to announce that an article I have written, titled Aspects of Domain Model Management, has just been published by InfoQ, my favorite online software development publication.

In the article I deal with many of the concepts I have discussed here in the blog, such as Domain Model Management (DMM), the Obese Domain Model Anti-Pattern and using Aspect Oriented Programming to apply runtime DMM infrastructure without making the domain model obese.

Even though I proof-read the article more than five times before sending the final version off for being published, seeing it again online I immediately discovered a number of bugs, in addition to my normal onslaught of spelling errors (including examples my complete and utter ability to distinguish between the words whether and weather) that the spellchecker won’t catch.

I will use this blog entry to list those mistakes that are not just spelling errors and update it as I discover new ones or they are pointed out to me.

I also didn’t see any link to the Visual Studio project with the code examples that I promised at the end of the article, so until that is fixed (or until someone tells me that I’m blind and the link is there) I’ll also keep a link to the downloadable code that accompanies the article here:

Download the code for the article here

I want to thank Niclas Nilsson for his invaluable help in writing this article – his feedback made me completely rewrite the article from scratch – and it turned out far better (and, ahem, longer). Thank you Niclas for not letting me go ahead and get that first awful attempt at an article published.

List of bugs:

List 2

The Persistence component should not contain any dirty flags.

Figure 6 (text above)

The text states that the person class inherits a dirty flag it doesn’t need, but it should be that it inherits a lazy flag that it doesn’t need.

Figure 7, 10, 11, 12

The EmployeeProxy class should have getName and setName methods (just like the PersonProxy class) overriding the ones on the Employee class in order to provide the interception.

List 4

The first property in the EmployeeProxy class misses its name. Currenlty it just says “public override string”, it should be “public override string Name”.

List 5

In the setter method for the Salary property in the EmployeeProxy class, the call to the OnPropertySet method on the dirtyInterceptor sends “this.name” as the second parameter – it should be “this.salary”.

All the calls to the lazyInterceptor send along a second parameter with the property name – but the interceptor method only accepts one parameter (and doesn’t need a second one). Thus all the calls to the lazyInterceptor should be changed to: “lazyInterceptor.OnPropertyGetSet(this);”

List 7

The Person class should no longer inherit from the DomainBase class (which has been deleted).