Modern BI

While I'm still an infant when it comes to using BI technology, being a software architect for many years has given me insight into the changing BI landscape. Facebook's Bryan Brandow explains how they opted for a more agile/exploratory model based on SQL and Tableau. Threre's a new player in town with a similar agile approach, Looker.  While weaker in visual aspects, Looker seems stronger in the modelling and reuse areas, allowing you to quickly define new reusable models based on existing data. Taking advantage of the power available in modern databases, Looker simplifies the BI stack in an appealing manner.  While we're still putting together our data warehouse, I'm very much looking forward to playing with Looker to see what it has to offer.

Dynamic JSON Deserialization

I recently ported some code from JsonFx to Newtonsoft's Json.Net to improve deserialization performance dramatically. JsonFx was deserializing to dynamic objects quite well, and when I tried to use Json.Net, I kept running into various problems.  The main difficulty was that it deserialized to JValue objects by default and before passing these into methods they needed to be cast or the data referenced through the .Value properties.  I wanted to just access the data directly as before without having to jump through extra hoops.  While I got close a few times, there was always some problem that kept confounding me.  I finally figured out the right incantation to deserialize to ExpandoObjects and I thought I'd share it here since I could not find this when I was searching.

JsonConvert.DeserializeObject(data, typeof(ExpandoObject), new ExpandoObjectConverter());

The following code which looks similar didn't work for me because the data didn't always represent an object.  It was sometimes a list or a primitive.

JsonConvert.DeserializeObject<ExpandoObject>(data, new ExpandoObjectConverter());

NuGet Circular Dependencies

Being heavy users of NuGet, here at the office we are frequently annoyed at the seemingly poor quality of the implementation. Between ridiculous performance (mainly of the local NuGet server), random breakages, and insane error messages, we spend far too much time managing the package manager instead of doing our jobs.

One of the most annoying errors we've encountered is when it claims to have detected a circular reference. This usually goes something like this:

Our Package #1 => Third Party Package #1 => Third Party Package #2 => Our Package #2 => Our Package #1

Good golly! It's a circular reference! No, wait... why is it that some random package off the internet depends on our own internal package? That's crazy talk.

We've seen two problems that can cause this madness.

1. NuGet is trying to install a dependency but the package for the dependency is not on the NuGet server. (This case has been confirmed.)

2. A package is missing from packages.config because NuGet randomly removed it at some point in the past. This again follows on the theme of the missing dependency. (We weren't as certain as to whether this was the exact cause because of related activity, but this is our best theory.)

If the NuGet folk happen to be reading this, regarding the performance of the local server I'd suggest caching the directory traversal to limit the IO activity during the query process. When you get lots of packages the IO to read them repeatedly during the use of the IQuerable seems significant. I'm sure I could go on and on about all the strange problems we've had, but I need to do some actual work, so I'll sign off now.

Hopefully this post saves a few hairs from being pulled out for somebody else out there.

Automatically Set Comment Visibility in JIRA

I struggled with setting the visibility of comments automatically in JIRA. I first tried the Jira Behaviours plugin but ran into the problems described here: https://studio.plugins.atlassian.com/browse/JBHV-158

Next, I tried using the groovy script runner plugin to set the security when a comment was created. After much banging my head against the wall, I finally got this to work. Unfortunately it was still e-mailing people before I had a chance to set the proper visibility.

The fix for that was to:

1. Remove all entries from the "Issue Commented" system event in the notification schemes.
2. Create a new custom event, which I fire in my custom handler for IssueCommented (see below).
3. Add notification scheme entries to this custom event instead, so people still get notified, but only after the comment visibility has been edited.

Here's the groovy listener I used to get the job done:

Cross-Origin Resource Sharing and Authentication

Cross-Origin Resource Sharing (CORS) is a nice newish technology that allows you to throw off the shackles of JSONP. To give you a taste, it allows these things:

• Proper error handling instead of waiting for some timeout.
• Ability to post arbitrary amounts of data.
• No helper libraries or crazy code required.
• Uses standard XmlHttpRequest (or XDomainRequest for IE8+).
• Basic Authentication (for non-IE browsers)

Feel free to ignore this paragraph, but before I proceed, I just wanted to talk at Microsoft about XDomainRequest. There was no need to introduce the entirely new XDomainRequest. You only needed to introduce restrictions on XmlHttpRequest in certain cases, just like all the other browsers do. I hope you at least add proper cross-origin support to XmlHttpRequest, if not fully deprecating XDomainRequest, in the future. Regarding XDomainRequest, I also find the lack of ability to set headers quite unfortunate. This basically means that authorization information needs to be passed as part of the core message, which forces a generic API provider to rip out the authorization from the message and reformulate it into a generic message for the normal API framework to process. Specifically, this makes it impossible to send OAuth headers. Note that while you could include the auth token in the url, this has the danger of leaking into logs.

Okay, let's get into the meat of this post. I was searching for a way to authenticate access to a REST API. Since the browsers I cared about supported authentication, I thought I would just follow the web standards and use regular authentication.

Besides being a massive pain to get going with WCF or an HttpListener, I later discovered what I thought was a serious security flaw. If you allow all domains to access the REST API (which is important) then browser caching of authentication is a serious issue. For example, you visit site A, authenticate to shared service S, and then visit malicious site B. Malicious site B accesses shared service S and since your browser has cached the credentials for S, it magically logs in and allows B to access the restricted content.

Well, it turns out the browser authors are smarter than me and they disallow authentication when allowing access to arbitrary domains. Herein lies the crux of design improvement #2 that I found myself wishing for. When you perform a request using XmlHttpRequest, you specify the credentials in the call to open(). Why couldn't the API simply not cache those credentials if accessing a CORS site with "*" for the domain restriction?

Okay, I've taken you through my journey. What's a REST API author to do for authentication? Well, since the browser is inept at managing to keep its data to itself, it seems we need to do it for it. As far as I can tell, cross-origin requests requiring authentication must pass some sort of authentication information with each request. I haven't fully explored the best way to do this, but there are a variety of schemes from signing the message to session tokens to sending the username and password as part of the request. I've been exploring the options here and I'll post more on my findings in the near future.

I'd be happy to hear if I missed anything, but I'm pretty sure I understand how lame the situation is. Well, at least it's not JSONP.

How Not To Make Passion Tea

I have taken a liking to Starbuck's Iced Passion Tea (with or without lemonade). Since I'm just a poor business owner, I thought I would buy a box of their Tazo Passion Tea and try to make some myself. Unfortunately, every time I steeped the tea, it ended up being too bitter.

After several attempts, I had the thought that it was my brewing technique that was causing the bitter taste. I had originally been pouring the boiling water on to the tea bag in an attempt to extract maximum flavour. Unfortunately, this seems to also extract maximum bitterness. When I made a cup of tea and deliberately avoided pouring the water onto the tea bag, the tea came out fine.

I'm looking forward to finding a good sugar/syrup/lemonade combination next.

Vancouver Eats

I recently went on a trip to Victoria and Vancouver for about a week. Here are some memorable foods from the trip.

1. Peaceful Garden Restaurant had great lamb cumin noodles. Their noodles are freshly made and cut right before cooking. I'm normally not a big fan of lamb, but I really enjoyed it.

2. Honey's Cafe in Deep Cove. Besides being beautiful and a great place to kayak, Deep Cove holds a secret treasure. Donuts. Go to the cafe and order one. They only have one kind, but to compensate, your donut is warm, soft, and delicious. Mmmm.

3. Last but certainly not least is Jimmy's Fruit Bar in the Lonsdale Quay. Being in business for a number of years, Jimmy has invented quite a variety of tasty and healthy drinks. My favorite while we stayed there was a drink called "Mr. Green Lifesaver." It's a delightful combination of mint, kale, spinach, pear, and mango, among other things. Tasty and refreshing. I'd highly recommend stopping by this friendly place for a great drink or two.

The Shrinking DLL Universe of NHibernate

There have been some interesting changes in the NHibernate universe recently. It used to be that you needed a whole bunch of DLLs and some extra config to do anything useful with NHibernate.

• NHibernate
• Iesi.Collections
• Antlr.Runtime
• Remotion.Data.Linq
• NHibernate.ByteCode.Castle
• Castle.Core

As of NHibernate 3.1, Antlr and Remotion have been ILMerged into the main dll, producing this picture.

• NHibernate
• Iesi.Collections
• NHibernate.ByteCode.Castle
• Castle.Core

As of NHibernate 3.2, there's now a default lazy proxy provider based on LinFu (thanks Philip). As well, this proxy provider is configured automatically, reducing a line of config required. This produces an even nicer picture for most users.

• NHibernate
• Iesi.Collections

Now we just need to replace Iesi.Collections with the .NET 4 ISet and we'll only have one dll to reference. I have no idea when that will happen, but it's an interesting thought anyways.

WPF Lazy Proxy Revisited for NHibernate 3.2

When hooking up to PropertyChanged events, some systems require that the same object that a handler was registered with be the same object that throws the event. With NHibernate proxying, this can be a problem. The external viewer normally hooks up to the proxy object, but then the actual implementation object is the one throwing the property changed event, using "this" as a reference, which is naturally different from the proxy. This behavior occurs because the proxying is not entirely transparent. Even though the user hooks up to the proxy object and expects events from the proxy object, the real object throws the events, messing with handlers that may be stored in dictionaries mapped by the sender. See my first post for other details on this same problem.

The original solution to this problem modified the Castle bytecode provider. Now, with NHibernate 3.2, there is a default provider built-in. Here is the new code with the fix for the PropertyChanged event. Using this code fixes problems with WPF and libraries like Obtics.

Linq Expression Evaluation

If you have a Linq Expresssion (such as a reference to a variable in the closure) or really any other self-contained fragment, you can easily evaluate it by turning it into a lambda expression. For example, I was trying to evaluate the content of a variable that was showing up as a ".x" field. To do so, I did the following:



 

NHibernate 3.0

NHibernate 3.0.GA has been released!  It has all sorts of goodies like QueryOver, logging abstraction, and tons more.  It also now has a built-in LINQ provider.  Unfortunately, the LINQ provider is still in a beta state.  Many queries work (as evidenced by the hundreds of working unit tests), but there are also still many queries it has trouble with.  If you'd like to help accelerate the development of the LINQ provider (or any other part of NH for that matter), here's what you can do:

1. File issues when you find bugs.  http://jira.nhforge.org/
2. Vote for issues.  Popularity is difficult to gauge if we don't hear back from people.
3. Provide test cases for those issues.  A complete test case is easier to digest than a tiny code fragment without context.  If the test can be used directly, the team member working on the issue can spend more of their time making improvements instead of trying to recreate a test.
4. If you're feeling ambitious, dig into the source code and try to work out a fix.

There's a great article on making tests for NHibernate here.  I've also included some different steps that might help you out.

1. Check out the trunk using something like TortoiseSVN (https://nhibernate.svn.sourceforge.net/svnroot/nhibernate/trunk/nhibernate)
2. Use ShowBuildMenu.bat to create the AssemblyInfo.cs files.
3. Load up NHibernate.Everything.sln in Visual Studio.
4. Create a test with the issue number in the NHSpecificTest folder of the NHibernate.Test project (NH2392 is a fairly simple example to copy from).
5. Use ShowBuildMenu.bat to see how to set up your database connection and run all the tests.
6. Create a patch (TortoiseSVN can help you with this) and attach it to the JIRA issue.

With your help, we will work towards a rock solid LINQ release that matches the tremendous power and reliability already provided by the NHibernate platform.

Interesting Changes in NHibernate 3.0

There are tons of new features in NHibernate 3.0. I'm going to highlight a few minor interesting ones from my view.

Client Profile Support
The .NET client profile is now supported out of the box.  Castle.Core also supports the client profile as well, so the Castle bytecode provider will work out of the box as well.  Great news for those who write desktop apps.

Logging Abstraction
In getting support for the client profile, log4net was an issue.  It hasn't been updated in ages and doesn't support the client profile itself.  This issue combined with other people's desire for a logging abstraction now means that NHibernate supports pluggable loggers.  You can use the NHibernate.Logging package to get support for libraries such as Common.Logging.  This also enables use of the excellent NLog framework, which is actively updated.

Polymorphism in Get and Load
Previously, Get and Load methods on the session required the exact mapped type.  Now, they allow you to specify a class or interface higher up the inheritance hierarchy, as long as there's only one match.  This will allow people to eliminate their own mapping layers when it comes to retrieving objects by primary key.  Importantly, this will make ActiveRecord work much better with interfaces.

Fixes for IdBag
The identifier bag had a bunch of bugs, especially ones that manifest themselves more when used in rich client applications.  If you had problems before, try it again.

ISQLFunction Argument Repetition
[Update: This will not be included in 3.0]

Though it's not in the trunk, I'm also excited about the potential inclusion of NH-2318.  This will let us write functions for dialects that have non-trivial argument patterns.  For example, a NullSafeEquals function could be written as (?1 IS NULL AND ?2 IS NULL) OR (?1 IS NOT NULL AND ?2 IS NOT NULL AND ?1 = ?2).  With the current code, such a function breaks because it repeats arguments.  The patch also deals with argument reording or removal, but it doesn't support ordinal parameters at the moment.  We'll see if this one happens to make it in.

All in all, NHibernate 3.0 is a very exciting release that many contributors have put a lot of hard work into.

Obtics

If you're working with .NET and any sort of observability, such as WPF bindings, you owe it to yourself to check out Obtics. It's an excellent library that turns an expression into an observable expression. It has full support for System.Linq.Enumerable methods and for arbitary expressions.

Using Obtics and a custom helper method, you can do things like:

public int AplusB
{
    get
    {
        return Getter(() => AplusB, () =>
            A + B
        );
     }
}

Now, whenever A or B change, a property changed notification is fired for AplusB.

Imagine now that the expression inside is not as simple as A + B but several lines of code involving collections, linq extension methods, and more. Obtics is able to automatically add observability to the entire complex expression.

Need to watch for changes to proerties in collections? Obtics has elegant ways of doing that too. It's a great library, very much worth checking out. In comparison to CLINQ or Bindable LINQ, Obtics has a focus on full expression support. Having full expression support is extremely powerful and Obtics seems to be an excellent implementation of it.

Pleasant Solutions

Starting a software company has definitely been one of the most challenging and enlightening experiences in my life. Owning a business takes a great deal of commitment and will to persevere. I am a partner at Pleasant Solutions, a custom software development house. We have a fantastic team and it's a joy to come to work every day and collaborate with these excellent individuals.

Specializing in custom software development is a difficult road on which to tread.  Although we are able to produce high quality tailored solutions for our clients, doing so profitably is a major challenge.  Thankfully, having excellent people allows us to adapt quickly to a broad range of tasks.  Developing certain infrastructure components has been very beneficial to us.  For example, we have a hiring system that allows us to screen our applicants in a more accurate and efficient way than reading resumes.  As another example, we significantly enhanced the open source ActiveWriter (with minor contributions to Castle.ActiveRecord and NHibernate along the way) to meet our needs.  In fact, we have continued to make improvements to our in-house tool to the degree that our developers need little knowledge of the nitty gritty of NHibernate in order to complete even complex business and data modelling tasks.  Building knowledge and experience into a tool has been very beneficial for us.  When we do our next project, we can bring along our past experience automatically, increasing quality and reducing costs.

We're still learning and exploring all the time, and we hope to continue to share our talent and passion for software development for years to come.

WPF and NHibernate Lazy Proxies

Using NHibernate proxies with WPF binding will likely leave you scratching your head at moments. There are at least two issues that can get in your way. The first of these is that objects with INotifyPropertyChanged implementations will not update when the property is changed. Even though the event is fired, the WPF value never updates itself. This is because WPF compares the source of the event against the object that it originally registered the event with. Since the source of the event is the actual implementation object hidden by the proxy, but the object WPF knows about is the proxy object, WPF ignores the event. The best way to fix this is to have the proxy object rewrite the property changed events. There is a detailed explanation and a simple implementation here, but be aware that it doesn't redirect events from the source but throws its own events when properties are set. The other issue you will sometimes run into is a strange null reference exception when setting the value of a dependency property. There is a bug in WPF where it doesn't properly handle the names of classes without any namespaces. Since the lazy proxy objects have no namespace, this bug pops up. The issue is fixed in WPFv4, but if you aren't able to upgrade, you can fix this by changing a line in Castle.DynamicProxy. In ClassProxyGenerator.GenerateCode, add the "Proxies." part to the following line: String newName = "Proxies." + targetType.Name + "Proxy" + Guid.NewGuid().ToString("N"); This should help anyone experiencing the error relating to DetermineWhetherDBNullIsValid.