Modern BI

While I'm still an infant when it comes to using BI technology, being a software architect for many years has given me insight into the changing BI landscape. Facebook's Bryan Brandow explains how they opted for a more agile/exploratory model based on SQL and Tableau. Threre's a new player in town with a similar agile approach, Looker.  While weaker in visual aspects, Looker seems stronger in the modelling and reuse areas, allowing you to quickly define new reusable models based on existing data. Taking advantage of the power available in modern databases, Looker simplifies the BI stack in an appealing manner.  While we're still putting together our data warehouse, I'm very much looking forward to playing with Looker to see what it has to offer.

Dynamic JSON Deserialization

I recently ported some code from JsonFx to Newtonsoft's Json.Net to improve deserialization performance dramatically. JsonFx was deserializing to dynamic objects quite well, and when I tried to use Json.Net, I kept running into various problems.  The main difficulty was that it deserialized to JValue objects by default and before passing these into methods they needed to be cast or the data referenced through the .Value properties.  I wanted to just access the data directly as before without having to jump through extra hoops.  While I got close a few times, there was always some problem that kept confounding me.  I finally figured out the right incantation to deserialize to ExpandoObjects and I thought I'd share it here since I could not find this when I was searching.

JsonConvert.DeserializeObject(data, typeof(ExpandoObject), new ExpandoObjectConverter());

The following code which looks similar didn't work for me because the data didn't always represent an object.  It was sometimes a list or a primitive.

JsonConvert.DeserializeObject<ExpandoObject>(data, new ExpandoObjectConverter());

NuGet Circular Dependencies

Being heavy users of NuGet, here at the office we are frequently annoyed at the seemingly poor quality of the implementation. Between ridiculous performance (mainly of the local NuGet server), random breakages, and insane error messages, we spend far too much time managing the package manager instead of doing our jobs.

One of the most annoying errors we've encountered is when it claims to have detected a circular reference. This usually goes something like this:

Our Package #1 => Third Party Package #1 => Third Party Package #2 => Our Package #2 => Our Package #1

Good golly! It's a circular reference! No, wait... why is it that some random package off the internet depends on our own internal package? That's crazy talk.

We've seen two problems that can cause this madness.

1. NuGet is trying to install a dependency but the package for the dependency is not on the NuGet server. (This case has been confirmed.)

2. A package is missing from packages.config because NuGet randomly removed it at some point in the past. This again follows on the theme of the missing dependency. (We weren't as certain as to whether this was the exact cause because of related activity, but this is our best theory.)

If the NuGet folk happen to be reading this, regarding the performance of the local server I'd suggest caching the directory traversal to limit the IO activity during the query process. When you get lots of packages the IO to read them repeatedly during the use of the IQuerable seems significant. I'm sure I could go on and on about all the strange problems we've had, but I need to do some actual work, so I'll sign off now.

Hopefully this post saves a few hairs from being pulled out for somebody else out there.

Automatically Set Comment Visibility in JIRA

I struggled with setting the visibility of comments automatically in JIRA. I first tried the Jira Behaviours plugin but ran into the problems described here: https://studio.plugins.atlassian.com/browse/JBHV-158

Next, I tried using the groovy script runner plugin to set the security when a comment was created. After much banging my head against the wall, I finally got this to work. Unfortunately it was still e-mailing people before I had a chance to set the proper visibility.

The fix for that was to:

1. Remove all entries from the "Issue Commented" system event in the notification schemes.
2. Create a new custom event, which I fire in my custom handler for IssueCommented (see below).
3. Add notification scheme entries to this custom event instead, so people still get notified, but only after the comment visibility has been edited.

Here's the groovy listener I used to get the job done:

Cross-Origin Resource Sharing and Authentication

Cross-Origin Resource Sharing (CORS) is a nice newish technology that allows you to throw off the shackles of JSONP. To give you a taste, it allows these things:

• Proper error handling instead of waiting for some timeout.
• Ability to post arbitrary amounts of data.
• No helper libraries or crazy code required.
• Uses standard XmlHttpRequest (or XDomainRequest for IE8+).
• Basic Authentication (for non-IE browsers)

Feel free to ignore this paragraph, but before I proceed, I just wanted to talk at Microsoft about XDomainRequest. There was no need to introduce the entirely new XDomainRequest. You only needed to introduce restrictions on XmlHttpRequest in certain cases, just like all the other browsers do. I hope you at least add proper cross-origin support to XmlHttpRequest, if not fully deprecating XDomainRequest, in the future. Regarding XDomainRequest, I also find the lack of ability to set headers quite unfortunate. This basically means that authorization information needs to be passed as part of the core message, which forces a generic API provider to rip out the authorization from the message and reformulate it into a generic message for the normal API framework to process. Specifically, this makes it impossible to send OAuth headers. Note that while you could include the auth token in the url, this has the danger of leaking into logs.

Okay, let's get into the meat of this post. I was searching for a way to authenticate access to a REST API. Since the browsers I cared about supported authentication, I thought I would just follow the web standards and use regular authentication.

Besides being a massive pain to get going with WCF or an HttpListener, I later discovered what I thought was a serious security flaw. If you allow all domains to access the REST API (which is important) then browser caching of authentication is a serious issue. For example, you visit site A, authenticate to shared service S, and then visit malicious site B. Malicious site B accesses shared service S and since your browser has cached the credentials for S, it magically logs in and allows B to access the restricted content.

Well, it turns out the browser authors are smarter than me and they disallow authentication when allowing access to arbitrary domains. Herein lies the crux of design improvement #2 that I found myself wishing for. When you perform a request using XmlHttpRequest, you specify the credentials in the call to open(). Why couldn't the API simply not cache those credentials if accessing a CORS site with "*" for the domain restriction?

Okay, I've taken you through my journey. What's a REST API author to do for authentication? Well, since the browser is inept at managing to keep its data to itself, it seems we need to do it for it. As far as I can tell, cross-origin requests requiring authentication must pass some sort of authentication information with each request. I haven't fully explored the best way to do this, but there are a variety of schemes from signing the message to session tokens to sending the username and password as part of the request. I've been exploring the options here and I'll post more on my findings in the near future.

I'd be happy to hear if I missed anything, but I'm pretty sure I understand how lame the situation is. Well, at least it's not JSONP.

How Not To Make Passion Tea

I have taken a liking to Starbuck's Iced Passion Tea (with or without lemonade). Since I'm just a poor business owner, I thought I would buy a box of their Tazo Passion Tea and try to make some myself. Unfortunately, every time I steeped the tea, it ended up being too bitter.

After several attempts, I had the thought that it was my brewing technique that was causing the bitter taste. I had originally been pouring the boiling water on to the tea bag in an attempt to extract maximum flavour. Unfortunately, this seems to also extract maximum bitterness. When I made a cup of tea and deliberately avoided pouring the water onto the tea bag, the tea came out fine.

I'm looking forward to finding a good sugar/syrup/lemonade combination next.

Vancouver Eats

I recently went on a trip to Victoria and Vancouver for about a week. Here are some memorable foods from the trip.

1. Peaceful Garden Restaurant had great lamb cumin noodles. Their noodles are freshly made and cut right before cooking. I'm normally not a big fan of lamb, but I really enjoyed it.

2. Honey's Cafe in Deep Cove. Besides being beautiful and a great place to kayak, Deep Cove holds a secret treasure. Donuts. Go to the cafe and order one. They only have one kind, but to compensate, your donut is warm, soft, and delicious. Mmmm.

3. Last but certainly not least is Jimmy's Fruit Bar in the Lonsdale Quay. Being in business for a number of years, Jimmy has invented quite a variety of tasty and healthy drinks. My favorite while we stayed there was a drink called "Mr. Green Lifesaver." It's a delightful combination of mint, kale, spinach, pear, and mango, among other things. Tasty and refreshing. I'd highly recommend stopping by this friendly place for a great drink or two.