Showing posts with label Intermediate. Show all posts
Showing posts with label Intermediate. Show all posts

Tuesday, December 8, 2009

Filtering LinqDataSource

Codes can be downloaded here

The RAD nature of ASP.NET Webforms makes it the sound choice for creating tools and utilities. I’m talking about small applications which aid development and are not intended for production. Among these features, the one that probably saves the greatest amount of time is data binding. I’ve tried all the data sources and with LINQ elegant syntax involving extension methods and lambda’s, I expected that with LinqDataSource, I can apply these new language features during filtering. I’m very disappointed to find out I have to deal instead with something I hated most – string literals!

image

The page I created is a log viewer which is nothing but a grid and a bunch of controls for the filter values. The illustration here is simplistic but the technique is not compromised in any way. The data is stored in a table with columns for the date the log was made, the level or severity of the message and of course the message. A grid will display this using LinqDataSource.

Config LinqDataSource 3 

Because it’s an ad hoc filter, I decided not to use the configurable WHERE parameters of the LinqDataSource. I thought I could add parameters as needed at run time. It is indeed possible and after some rummaging, I found out that the best place to do this is in the  LinqDataSource.Selecting event. In this event, the sender is a LinqDataSourceView which exposes a Where string property. This accepts the body of the lambda expression that one usually put in the Where extension method. The only catch: it’s string. Beautiful! The tough part is when dealing with dates because you have to use DateTime utility methods and all this without the benefit of Intellisense. Shown below are the pertinent codes for the sample app you can download for this post.

image

Tuesday, November 17, 2009

Temporal Data: Considerations, Conventions and Cheats

I’m not an expert of temporal database, its concepts and principles. The techniques I discuss here are the ones I employed to an application during my early days of programming. For exhaustive information on this subject, you may consult the works of Richard T. Snodgrass or C.J. Date. It’s so easy, just Google them. ;)
Temporal Granularity
The first thing to establish in a temporal database is the shortest length of time by which a change in the data is considered significant. This is commonly referred to as the "temporal granularity”.  In my application, I assumed it to be by “day” but SQL Server didn’t have any DATE-only data type until version 10 so this limitation was taken into consideration during the development. A function was created to remove the time portion of a DATETIME.
image
Figure 1 – A function to remove the time portion of a DATETIME
Index
If you isolate the columns to be tracked in another table, the new table should have a unique clustered index on the combination of the foreign key and the start date time. An example of this is the operation history table which tracks changes of the rate of an operation. In that table, the foreign key and start date combination is part of a unique clustered index to ensure that there is always one valid rate per given date. It also wouldn’t hurt if a check is employed to make sure that the end date is always after the start date.
image
Figure 2 – Table containing temporal data should have unique clustered index on the FK and start date/time
Querying Strategy
Querying a version of data in a temporal table always involves a date parameter indicating the version to retrieve. This date should fall between the start and end dates of at most one row for every id. The id here refers to the foreign key pointing to the non-temporal parent table. To make this clear, consider the following transactions:
  1. Operation “Button Attachment” with rate of 0.35 is inserted on 10/28/2009
  2. Rate for operation “Button Attachment” is changed to 0.37 on 11/01/2009
These insert 3 rows: 1 for the operation table and 2 to the operation history. The former is not pertinent to the discussion because it does not contain temporal data. In the latter, the rows inserted are highlighted below:
image
Figure 3 – Sample rows showing the current and previous version of a temporal data
If the user asks for the rate on 10/30/2009, obviously she would retrieve 0.35. The query is straightforward as long as the date fall between the transaction dates. What would she get then if she asks for the rate on 11/01/2009? Again, this is where business should come to the rescue. The business should establish the convention employed during the closing and opening of two contiguous periods, which happens when the data is updated. The convention can be determined by simply asking two questions:
Given an update on date n
  • What should be the end date of the previous period?
  • What should be the start date of the current period?
The answers determine the insert and update strategy to employ. If for the first question, Business says n-1, then it also implies that the answer for the second question is n. It can also be an n for the first and n+1 for the second. Important thing is that they should not overlap. After the convention is established, the BETWEEN operator can be safely used in the query as shown here:

SELECT o.name_vc, h.labor_sm FROM t_operation o
JOIN t_ophist h ON o.operation_id = h.operation_id
WHERE o.operation_id = @operation_id
AND (@versionDate BETWEEN h.start_dt AND h.end_dt)
Figure 4 – The most correct way of querying temporal data
In Figure 3 however, one can see that in my application, the end of the previous and the start of the current periods are the same. This is because back the, I didn’t know about BETWEEN operator. To achieve the same effect, I had to use inequalities operator and lines like these were all over the place:

start_dt <= @versionDate AND @versionDate < end_dt
Figure 5 – A condition used in the application in lieu of BETWEEN

Thru that filter statement, I effectively established the convention of n on end of current and n+1 on the start of the new version
Data Modification Logic
One interesting aspect of temporal data is that it changes the conventional semantics of data modification.  Update can be any of these 3 cases: update plus insert, update only, delete and update. A delete in the other hand is merely an update on one column. Finally, an insert may only occur in the temporal table and not in the parent (non-temporal) table. Now it should be clear why dealing with temporal data is definitely not a walk in the park.

When a temporal data in my application is updated, the start date of the current version is compared to the date of the transaction. The first case in update happens if the transaction date is not the same as the start date of the current version. The current version is closed or invalidated and a new version is inserted. Invalidating a version is just providing a valid value to the end date depending on the convention used. The “Button Attachment” transactions is an example of this case. The update is later than the start date of the current version therefore a new record is inserted and the current version is invalidated with the date of the transaction.

The second case happens when the transaction occurs on the same day as the start date of the current version. A delete and insert in this case is an overwork because all that is needed is an update on the rate (labor_sm). If we use the “Button Attachment” as an example, this is the case when the rate is raised again to 0.38 on 11/1/2009.

The third case is a special case of the second. This happens when the new value is the same as the value of the previous version. The previous version is the one preceding the current. In the “Button Attachment” example, this happens when the rate is changed back to 0.35 on 11/01/2009. This is like not updating the 10/28/2009-11/01/2009 version at all. So the course of action is to delete the current and open the 10/28/2009-11/01/2009 version.

image
Figure 6 – An stored procedure illustrating the different cases involved in updating a temporal data
As mentioned before, there is no actual delete in a temporal database. Delete is achieved by simply closing the current version on the temporal table. In the example, if “Button Attachment” is deleted on 11/01/2009, the only action taken is closing the current version. The record “Button Attachment” in the operation table is left intact. This means that during insert, it’s possible that the data is already in the operation table. In such a case, an insert to the operation history for the new valid version of the said data does the job.
Cheats
Later during the development of the application, it became clear that most of the queries in the frontend involved only the current data. I was fully aware of the performance implication innate to temporal queries like that in Figure 4 so I decided to employ a cheat. Instead of BETWEEN, I used = operator to get the current version since a convention is already established that end date for current should always be 12/30/5000. This also means that end date was no longer null-able and was included in the unique clustered index together with the FK and start date.

Another cheat that I employed later in my other projects is what I call the “reverse relationship”. It still uses = but instead of end date, the parent table contains a foreign key of the temporal child table. This also necessitates having a surrogate primary key in the temporal table. Usually, like in Figure 2, the temporal table can use the combination of the FK and the start date as a composite PK but in the case of reverse relationship, this composite key becomes a bulky FK in the parent table. A lean and faster approach is to have a surrogate key as FK like the one illustrated in Figure 7. Of course, it doesn’t have to be a constraint; a column pointer would suffice but careful check should be done to avoid pointing to a non-existent temporal data. The “fast” factor comes into play in there is a very high volume of data in the temporal table. A slight performance benefit can be squeezed out from querying integer instead of a date.
Reverse relationship
Figure 7 – A sample parent-temporal child table with “reverse relationship” to speed up temporal query
I’ve also seen temporal table that has flag column to indicate if the row is current. Like the reverse relationship, any technique that involves additional column to convey a semantics also open the database to potential synchronization problems. For example, it’s possible that a flaw in the update logic might insert the wrong FK in the parent table or flag the wrong row as current.

I’m sure there are still more “cheats” or techniques out there. A modeler can employ even the most unconventional but it shouldn’t matter as long it’s justified. These kind of practices deviates from the standard so proper documentation of these “deviation” is also a must.

What’s Next

Next post will be about the preliminary efforts I had when translating the VB6 version of the application to .NET version. I’ll discuss the overall architecture and motivation of the approach I chose.

Sunday, November 1, 2009

Temporal Data

In the first installment of this series, I mentioned that I unnecessarily complicated the ApparelFactory app by introducing temporal data, thus delving myself into the world of temporal database. A temporal database differs from an ordinary database in that it stores different version of data based on their validity throughout time. Other people simply refers to these versions as “history” of the information. A piece of data is said to be “valid” on a certain time if it is consider to be true at that time.  
“Querying a particular version of a temporal data is not straightforward because you need to consider the time component.”
It’s easy to spot a database which contains temporal data. You usually see two time columns in a table indicating the start and the end during which the data is considered valid. The granularity dictates the data type used. For example, if you don’t consider changes within a day to be relevant, then you can use DATETIME ignoring the time portion. SQL Server 2008 has a DATE data type well-suited for this need. If changes within the year is not relevant, you can use INT for both the start and end years. The granularity is usually dictated by the business but in the case of the ApparelFactory, I simply assumed that changes within the day is not relevant, thus I have a tables like the one below where start_dt and end_dt are the start date and end date respectively.
image
Figure 1 – A table for storing temporal data
Most of the time, users are only concerned with the current valid version of the data. Past versions are of interest only for analysis and evaluation of trends, say the price fluctuation of certain commodity. They may be viewed only through reports and some other data warehousing tools. Querying a particular version of a temporal data is not straightforward because you need to consider the time component. For example, in the t_allowance table, if I were to write a flexible query to retrieve any version an employee’s allowance by date, the WHERE statement would have looked like this
WHERE employee_id = @employee_id AND (@referenceDate BETWEEN start_dt AND end_dt)
Figure 2 – Flexible WHERE statement for a temporal data
A query involving BETWEEN operator is not the most efficient so coders usually cheat when writing one for retrieving the current version. They use = instead, but this presupposes that there is already an established convention in determining the current version. The most common is to make the end date nullable and any row with a null value is considered current. In the application, I used the another variation which sets the end date to a very unrealistically high value. This has the advantage of not requiring you to convert the null end date to a real value when using BETWEEN. This is because using BETWEEN on a null date always evaluates to false.

image
Figure 3 – Sample query utilizing equality operator instead of BETWEEN for temporal data
In a true temporal database, every change in the data is recorded. This is not practical since most of the time, only a subset of the columns are worth tracking. Only the columns  which trends are worth analyzing are tracked. These are then separated in a table together with  the start and end time columns. In the apparel app, I made sure that user can always see the history of the rates of every operation as well as their usage in a style (what operation is involved in a style at a certain point). The queries would have been complicated had I not employed some “acceptable cheats” in my table design. I’ll discuss more about these so-called cheats in the future installment of the series.

image
Figure 4 – Column which need to be tracked are usually taken out and made into a separate table
Temporal data are useless if the coder doesn’t employ an intuitive way of presenting them to the user. All these data should be available to reports of course, but users also appreciate it if they could see them instantly without resorting to reports. This way, they don’t have to be inundated with so many information when all they want is, let’s say the rate history for an operation which rate the company has just decided to change. In this connection, the ability for a user to select a date as a parameter during viewing of the data is very integral when designing the UI. A coder can simply throw in a DateTimePicker or Calendar web control. If the user prefers the actual change dates, then a list is a good choice. This approach involves another query but it always ensures new results during every query. It also presents the user with the exact number of changes; something not available with the date time picker / calendar approach.

In the application, the list approach is used as seen in Figure 5. The number of current operation for the style is 3, one is added sometime on 11/01/2009. The rate of one operation also changed on the same date, thus changing the current rate of the style.

Style history 1
Figure 5 – A UI displaying historical changes on data membership

The addition of another operation and the change in rate of another one can be seen in Figure 6. Since only the rate is tracked for the operation, it makes much more sense to simply display the history right away.

Operation view
igure 6 – A UI displaying historical changes of a single column
Being able to work on temporal database is really a fulfilling experience for a coder, let alone for a novice. A coder feels that his application adds more value to the business because it ensures all the data are there and easy for the viewing. The flipside is, it might not be needed at all and it’s just another unnecessary complexity to an already delayed project. It wouldn’t hurt if the client is asked first because the last thing a coder needs is a flawless feature that no one even dares to use.

Next, I’ll discuss the adjustments, conventions and the cheats I employed.

Friday, October 16, 2009

Logging the Stack Trace

Codes can be downloaded here

Stack trace is very important in locating exceptions because unlike the Message property, it can pinpoint the exact location of the line that throws the exception. Unfortunately, developers seldom log the stack trace for 2 reasons:

  1. They can be very long which in turn requires a large storage capacity
  2. The information don’t help that much because the methods are listed in reverse order.

The size of the stack trace depends on where the method that throws the exception is located in the call stack. Take for example adding of record functionality in this application:

image

The function has a very shallow stack which is initiated by a button click and terminates in the database. This is illustrated by the call stack diagram below:

Button click in Default.aspx  -----calls----> Entity Framework Model -----calls------> Database

If a line in the “button click” method causes an exception, say a database integrity violation,  the first method in the stack trace is not not the “button click” but the one that detects the violation which is shown here:

at System.Data.Mapping.Update.Internal.UpdateTranslator.Update(IEntityStateManager stateManager, IEntityAdapter adapter) at System.Data.EntityClient.EntityAdapter.Update(IEntityStateManager entityCache) at System.Data.Objects.ObjectContext.SaveChanges(Boolean acceptChangesDuringSave) at System.Data.Objects.ObjectContext.SaveChanges() at WebApp._Default.btnException_OnClick(Object sender, EventArgs e) in C:\Users\Vanni\Documents\Visual Studio 2008\Projects\StackTraceSplit\WebApp\WebApp\Default.aspx.cs:line 28

For the developer searching for the bug in a production environment, the last method in the trace is the most important because that is where he’s going to start the search. To make his life easy, one can write a extension methods for System.Exception that can reverse the order of the stack trace and retrieve only a portion of it, usually the ones which the developer has access to. Extension methods are user-defined methods that can attached to an object which implementation is off-limit to developers. The method should adhere to the following guidelines in order for it to qualify as an extension method

  1. It has to be static
  2. It has to have a return value
  3. The first parameter should be in the form “this <object to extend> parameter value”

The extension methods for the predicament discussed may look like the ones below. Note that it’s a good practice to specify the class as static because this ensures that all methods should be static too.

 image

The first method splits the entire stack trace into arrays by using the characters “ at “ as the separator and reverses the order. The second is an overload which accepts a number of stack to return, starting from the top. The third method returns only one big chunk of the stack trace and is intended for logging.

When running the following codes,

image

a portion of the result would look like this:

image

Now that will definitely make a debugger’s life easy!

Wednesday, October 7, 2009

“Nullable object must have a value.” – could be a sign of something serious

At first glance, the InvalidOperationException with the message “Nullable object must have a value.” is probably the most senseless of all .NET exceptions. This is because the message is very misleading when one takes it by its face value. The most logical conclusion a reader can derive is that it’s something about failing to assign a non-null value to a Nullable<T> object. Microsoft could have used a much better verbiage because what’s actually triggering the exception is an attempt to perform an action to the Nullable<T> object which value happens to be null. In other words, it’s about an invalid action, not value.
image
Figure 1 – A message that can leave you dumbfounded
Unfortunately, the capability of Visual Studio in detecting null nullable is limited only to uninitialized fields. If the nullable object is a property, then all you can do is sit and wait for the bomb to go off. In Figure 2, the properties SampleClass.NullableBool and Program.NullableBool are never checked by Visual Studio whereas _nullableBool cannot escape the scrutiny.
image
Figure 2 – Visual Studio can help detect only uninitialized nullable fields
Figure 3 shows the most common mistakes that trigger the exception. The first case is when the Value property is called to extract the encapsulated native type. The second case is quite common among new programmers where instead of calling Value, a cast is performed on the nullable object.
image
Figure 3 – Be on the lookout for lines similar to these
Bear in mind though that calling ToString() on a null nullable is perfectly legal as long as it’s initialized. It simply yields a blank which is what exactly we want in the UI.
image
Figure 4 – It’s safe to call ToString as long as the nullable is initialized

There Could Me Something More

When you’re done cursing Microsoft, do yourself a favor and go back to your codes. The message could be a sign of a more serious type of bug lurking somewhere. This is because it’s possible that nullables are used in boolean conditions without the benefit of a prior nullability check. If this is the case and the coder does not call the Value property, the compiler does not complain at all. It simply evaluates the condition as false!
image
Figure 5 – A null nullable is always evaluated as false
In the business rules of your application, it might be that a null is equivalent to a false and in this case, you could care less about codes similar to Figure 5. However, the fact that 3-value logic is already allowed to permeate in the application also means that null deserves a separate treatment altogether. A null could mean “I don’t know”, “not yet”, or “maybe”.
image
Figure 6 – This block can wreak havoc in your application.

Defensive Coding

Nullable<T>.HasValue can be used to check if the value is null. You usually use this whenever a null value is treated differently as illustrated in the snippet below:
image
Figure 7 – Using HasValue prevents exception and aids in the 3-value logic evaluation
In the rare cases where a null means false or any valid value of a type, one can use the coalesce C# operator ?? as shown here:
image
Figure 8 – The coalesced operator is handy when null value can be interpreted in some other way
In the next release of .NET, I hope we could get a much more meaningful message, something like “Attempting to perform action to a Nullable<T> is not allowed when its value is null.” Whatcha think?

Sunday, September 20, 2009

My First Battle

A few months back, I got a notice from Yahoo! that it’s was about to pull the plug on Briefcase, its free online storage service and that I needed to transfer all the files. There, I found a zip file of the very first industrial-strength project I developed almost 9 years ago together with some friends. When I got a copy of Visual Studio 6 from a friend, I decided to look into the codes again and what I discovered are traces of struggle, perseverance, and naiveness that shaped the coder that I am today. Every project is a battle. This is the first one and I had only the slightest idea of what I got myself into.

The project is a VB6 client-server application for an apparel company. It was used to compute wage for factory workers based on the work items (known as job) they’ve done for the week. It was originally a college project but the primary stakeholders, realizing the huge return of investment from automated process, tasked them to continue working on it for a price. Well, we didn’t really know the price at that point but the idea of making money out of the craft you’re trying to learn from school was just too enticing. I joined the team when it was realized that MS Access was no longer up to the job. In my case, my enthusiasm was further bolstered by the fact that I was still beginning to learn SQL Server.

The project became an eye-opener to the real nature of the craft I chose. I didn’t graduate with a computer degree but I decided to pursue programming thru self-thought because it’s the only way I can satisfy the “control freak” in me. I thought it would be fun, after all it’s about doing what you really love, right? Well, nothing could be further from the truth. It turned out to be a nightmare; capable of turning anyone into a zombie due to several sleepless nights and skipped meals. The same is the nature of software development as I experience nowadays but the big differences is that we were still naive then. We lacked the experience and knowledge about good software development practices. Exacerbating the situation were the comparatively primitive technologies we’re using which always compelled us to make compromises in order to achieve the best solutions. This post, and the ones hereafter, will explore the mistakes I made and how they influenced the subsequent projects I had. I will also discuss the by which they should be implemented using the most current technologies.

The Model and the Names

Virtually every business application deals with data so it just makes perfect sense if I start with the data model. I can say that I designed about 90% of the data model. Another member took care of the remaining 10% for an auxiliary applications which he solely developed. The first thing you would probably notice is the application of Hungarian Notation and underscores. I learned this style and the naming convention, which you will see momentarily, from the book I was reading at that time. As a beginner, you tend to be dogmatic and don’t even dare to ponder if what you’re imitating is actually right. I later found out that this style is not just hard to read but totally unnecessary.

There were more tables than what were actually needed. This is because I maintained history on some data which I thought would be valuable for auditing and report generation. This is actually a big mistake. Queries involving temporal data are not straightforward because you’re always dealing with composite unique keys involving dates. Besides, the stakeholder did not ask for it - I just assumed it. The adjustments we made just to cater for temporal data, significantly delayed the delivery of the project. I was guilty of that.

“As a beginner, you tend to be dogmatic and don’t even dare to ponder if what you’re imitating is actually right.”

image

The model included extraneous tables.

I did not use the IDE to create my database objects. Instead, I painstakingly wrote the script for the entire database! For a beginner, it’s a great way to learning SQL but it’s a bad practice simply because it takes so much time. Companies don’t hire you because you can write an entire script of a database. They hire those who can achieve things within the shortest period of time so that means being adept with tools.

image

Sample script for a table, complete with constraints.

“The adjustments we made just to cater for temporal data, significantly delayed the delivery of the project. I was guilty of that.”

Later in the development, I’ve had enough of Hungarian Notation and this couldn’t be more evident than the contact number column. From a book I referenced, contact numbers were stored as integer and I just blindly followed that. But having it so requires parsing and formatting in the UI. Worse, changing its data type also means making sure the procedure or views that used it were modified because the column name has changed. Coupling is the biggest headache from Hungarian Notation especially that time when there was still no refactoring tools. Another lesson I learned from this is that a column, even though they can be stored as numbers, should never be numeric unless it's being used in some kind of computation.

image 

Some columns did not use Hungarian Notation anymore

What’s next

this point, it’s should be clear that I just scratched the surface of one laborious undertaking. In the next installment of this series, I’ll discuss the complexities involved in dealing with time-aware data.

Tuesday, June 30, 2009

OUTER JOIN in LINQ

I’m quite disappointed to find out that LINQ has an OrderByDescending() and the “VB-ish” ThenBy() but seems to forget a dedicated extension method for OUTER JOIN. Although not as straightforward as I wanted it to be, SelectMany offers a working solution.

image 

For illustration, let’s consider the very naive model below. A left join from Child to Parent should yield 2 rows, with the second row having null values on the Parent columns.

image 

Dragging the tables to your LINQ-to-SQL canvass produces the model below. From this we can issue our statements.

image

The solution uses “statement” syntax, not the SQL-like syntax you always see when someone touts about LINQ features. To help with the breakdown of the LINQ statement structure, I included the SQL syntax and highlight the equivalent parts.

image

Part labeled 1 is the projection statement with the LINQ version utilizing anonymous type. Notice that type inference compels me to convert the ParentId to nullable type. Not doing so would have led the compiler to use Int32 as the type for ParentId. If this happens, an exception would occur for the second row because the value assigned to that property would be null.

Part labeled 2 qualifies the join. The object from which SelectMany is called returns all the rows. This is exactly what we wanted with the Child table. Also notice that we made sure that the qualifier columns from both table have the same type. In order to achieve this, the Value property of the Nullable<Int32> is called from Child.ParentId. You can also achieve the same by converting the other side into nullable as shown here:

image

You also need to call the DefaultIsEmpty() on the right table, in this case the Parent table. If you don’t call this method, the compiler would just ignore those rows with null values in the Parent table, effectively reverting to an INNER JOIN. Weird isn’t it?

Running the small snippet gives exactly what we wanted.

image

Saturday, June 27, 2009

Taking Care of Time Zones

When creating an application accessible anywhere around the globe, special attention must be given to data involving date and time. A mere DateTime data type is not enough and sometimes, could be a dangerous proposition. The system should also be aware of the time zone. Fortunately, this is very easy today that SQL Server 2008 has already a data type specifically for this requirement – the DateTimeOffset.

Let’s consider a web application that allows viewing of historical data with regard to the status of a task. A designer who tends to skimp on date and time data analysis would come up with the “wrong way” model similar to the left part of this illustration:

image

Query for historical data usually involves passing the date time when the query is requested and then qualifying this date time BETWEEN the start date and end date columns. To signify that a record is the latest version, the end date is set to null or unrealistic high value like ‘12/30/9999’. This guarantees a match if the current date is passed. Applying this technique in our query for the wrong model, the following scenario would fail:

A user from Manila, Philippines updates the status of Task 1 on 06/26/2009 0900 local time. Two hours later, a user from Tampa, FL tries to view the latest status updates.

Here is the content of our database after the update in Manila

image

The scenario fails because in Florida, it’s still 06/25/2009 11:00PM and as far as the system is concern, that is earlier than the update date:

image

The DateTimeOffset data type of SQL Server 2008 makes it easy to avoids this pitfall. Prior to SQL Server 2008, the solutions would usually involve saving the hour offset in the database and use this to manipulate the timezone-agnostic date time in the UI. We applied this new data type in the right side of the model - the “Right Way”. For this model, I opted for LINQ using LinqPad. I find LinqPad to be a very effective prototyping tool and maybe you should give it a try. So back to our discussion, the codes below first simulate the transaction from different time zones. After that, we use the generated timezone-aware dates for the query. As you can see, our parameter qualifies for the condition, as expected, even though the date parts are very far apart. Because of the offset value in the dates, SQL Server correctly computes and determines that Tampa time is actually just 2-hour later.

image

The DateTimeOffset is really a very valuable addition to SQL Server. It addresses a small but significant shortcoming SQL Server in an era of outsourcing, e-commerce and cloud-computing. I myself is already wary on using DateTime because I just don’t know when a pathetic application of mine would go global. Oh yeah! ;)

Sunday, April 19, 2009

Using Table Data Type for Variable-length Parameterized Search

The table data type is introduced in SQL Server 2008 to address the clamor of the community for an efficient means of inserting and updating multiple rows in the database. What used to be a task involving multiple calls to the database can now be achieved in one swoop. True to this objective, it’s no surprise that almost all discussion of table data type usage deals about this topic. Another one which is rarely discussed is on how table data type addresses one of the most volatile code construct in data-driven application – the variable-length parameterized search construct. This is the code construct for a search wherein you allow the user to specify 1 to n number of values for filtering a certain column. The user interface usually resembles the one below:

image

The interface above is obviously simplified. Most of the time, it involves a filter for almost every column in the result. The assumptions here are that the application does not cached any fetched data and the team has been living too long inside the cave that they haven’t heard about LINQ yet. Before SQL Server 2008 and LINQ, there are two approaches to achieving this, each has it’s own pros and cons.

The first approach is maintenance-free but opens your database to a possible SQL-injection attack. It involves a dynamic query which is composed of multiple strings. One of these strings is passed from the frontend application because it contains the values for the filter. The structure of the stored procedure may look like this:

CREATE PROCEDURE FilterDynaQuery @filterValues  VARCHAR(8000) AS
DECLARE @select VARCHAR(2000)
,@sort VARCHAR(500)
SET @select = ‘SELECT col1, col2, col3, … FROM Table1’
SET @sort = ‘ ORDER BY col1, col2, col3,…’
EXEC sp_executesql (@select + filterValues + @sort)

The @filterValues must contain all he hardcoded values from the frontend. The coder must make sure that regardless of the intention of the frontend, the concatenation of the all the components of the query should be valid. The argument passed should conform to the construct “ WHERE <column> IN (val1, val2,…valn)” like the one shown below
‘ WHERE col1 IN (1,5,8,4)’

The second approach ensures the safety of the database but couples the codes to the number of possible values in every filterable column. If the business adds or removes value allowable for that column, then the signature of the store procedure and the front-end codes should also change. For example, if there are 3 allowable values for that column, you need 3 parameters on for that column. Here’s how the procedure would then look like:

CREATE PROCEDURE FilterExplicitParams @value1 INT, @value2 INT, @value3 INT AS
SELECT col1, col2, col3 FROM Table1 WHERE col1 IN (@value1, @value2, @value3) ORDER BY col1, col2, col3

If you have many possible values for the column to be filtered, using this approach would surely bloat the codes. We all know that the more codes you write, the greater the chance for bugs and delays. Most coders - me included - would rather compromise security a bit, in exchange for easily maintainable codes. This is why the dynamic query is much preferred.

The New Approach

The table data type provides you the benefits of both approach without the compromises. However, be aware that RAD tools have no full support for queries with table-valued parameter, at least for now. In ASP.NET, you cannot use the SqlDataSource directly because you will not be able to fetch the schema of the result. ASP.NET requires the schema to configure the data control to which the data source is bound. This means executing the procedure using the dialog shown below. The problem is that it cannot accept the .NET types mapped to a SQL Server table type.

image

With ObjectDataSource this is not an issue because the tool doesn’t have to query the database for the schema. The quickest way to create an object for the result of our procedure is to make use of typed DataSet. Yes, this somewhat misunderstood and easily abused component is a boon for this kind of situation in a data-centric application. The stored procedure which will be discussed momentarily would become the source for the select command of the TableAdapter. If you’re new to typed DataSet, an excellent walkthrough can be found here.

For the sample application, we retrieve all the current employees from the AdventureWorks database. The tables involved are shown below.

image

Since the data is temporal, I decided to create a view which would contain only the current set of employees.

CREATE VIEW HumanResources.CurrentEmployee AS
SELECT e.NationalIDNumber, c.LastName, c.FirstName, c.MiddleName,
d.DepartmentID, d.Name AS Department, e.EmployeeID
FROM HumanResources.EmployeeDepartmentHistory h
JOIN HumanResources.Department d ON h.DepartmentID = d.DepartmentID
JOIN HumanResources.Employee e ON h.EmployeeID = e.EmployeeID
JOIN Person.Contact c ON e.ContactID = c.ContactID
WHERE (h.EndDate IS NULL)
The stored procedure accepts a table-valued parameter of type IntId. This is just a one-column table type that can be used to filter a integer column, just like the Employee.DepartmentId. Filtering a column is achieved by a join to the table or view to be filtered; in our case the CurrentEmployee
CREATE TYPE IntId AS TABLE
(
[Id] INT NOT NULL PRIMARY KEY
)
GO
CREATE PROC FilterEmployeeByDepartment
@DepartmentIds IntId READONLY
AS
SET NOCOUNT ON

SELECT e.EmployeeID, e.NationalIDNumber, e.LastName
, e.FirstName, e.MiddleName, e.Department
FROM HumanResources.CurrentEmployee e
JOIN @DepartmentIds ids ON e.DepartmentID = ids.Id
GO
Examining the parameter of the Select command in the TableDataAdapter would reveal that the mapped .NET object for the SQL table type is System.Object.

image

Don’t be dissuaded by what the editor seems to suggest. The only types you can use according to the documentation are DataTable, DbDataReader and IEnumerable. But even without having read the documentation, you would have probably used DataTable anyways; it’s just too obvious. Among the 3, IEnumerable still eludes me and I’m still struggling to get my first shot with it. I could care less; for now DataTable is perfectly fine for the job.

Setting up ObjectDataSource with typed DataSet is fairly easy. You can use the TableAdapter directly as shown below. In the next step, Visual Studio lets you specify which method is mapped to which CRUD method. With the sample application, the Get method of the adapter is used for the SELECT and the rest of the CRUD are just left blanks. The argument for the parameter is set to get its value from the Session.

image

The rows from the HumanResources.Department table should be fetched too. In the sample application, a SqlDataSource is used for this. A list box is bound to this source while a grid view is bound to the object data source discussed a while ago. There is no code necessary to populate the list box. As for the grid view, a dummy table is supplied during the loading because null is not yet supported by the ObjectDataSource with table-valued parameter. In the Find button click, the table is populated accordingly prior to passing it to the data source.

protected void Page_Load(object sender, EventArgs e)
{
if (!IsPostBack)
{
DataTable tbl = CreateDeptFilterTabl();
Session["DepartmentIds"] = tbl;
}
}

protected void btnFind_Click(object sender, EventArgs e)
{
DataTable tbl = CreateDeptFilterTable();
GridView1.Caption = "Employee(s) from ";
foreach (ListItem item in ListBox1.Items)
{
if (item.Selected)
{
tbl.Rows.Add(item.Value);
GridView1.Caption += (item.Text + " ");
}
}
Session["DepartmentIds"] = tbl;
DataBind();
}

DataTable CreateDeptFilterTable()
{
DataTable tbl = new DataTable();
tbl.Columns.Add("id", typeof(int));
return tbl;
}

Codes

You may download the codes here. If you don’t have AdventureWorks database, you can download it here. Be sure to change the connection string before you run the application.

Friday, April 10, 2009

SqlUserDefinedAggregate.IsNullIfEmpty: Does it work?

No. At least for me. I haven’t made this one work at all and I always have to resort to a dirty workaround. According to MSDN documentation, setting this property to true should direct your aggregate to return SQL-null when applied to an empty table which is exactly what you wanted. However, this is not the case. Let’s take a look at this naive example of a UAG which simply concatenates the string value of a column:

[Serializable]
[Microsoft.SqlServer.Server.SqlUserDefinedAggregate(
Format.UserDefined, MaxByteSize=8000, IsNullIfEmpty=true)]
public struct ConcatStr : IBinarySerialize
{
SqlString _concatenatedStrs;

public void Init()
{
_concatenatedStrs = null;
}

public void Accumulate(SqlString value)
{
if (value == SqlString.Null)
_concatenatedStrs += value;
}

public void Merge(ConcatStr group)
{
_concatenatedStrs += group.Terminate();
}

public SqlString Terminate()
{
return _concatenatedStrs;
}
#region IBinarySerialize Members

void IBinarySerialize.Read(System.IO.BinaryReader r)
{
_concatenatedStrs = new SqlString(r.ReadString());
}

void IBinarySerialize.Write(System.IO.BinaryWriter w)
{
if (_concatenatedStrs != SqlString.Null)
w.Write(_concatenatedStrs.Value);
}

#endregion
}

There’s nothing fancy about the codes and its implementation is pretty much straightforward. The quirk is, you don’t get a “NULL” if you apply this to an empty table as you see here

image

The workaround is simple. You need a flag that you can check if the Accumulate() method has been called or not. Accumulate() is called for every row in a table so no-call on this means the table is empty. Here’s the modified codes for our string aggregator

[Serializable]
[Microsoft.SqlServer.Server.SqlUserDefinedAggregate(Format.UserDefined
,MaxByteSize=8000, IsNullIfEmpty=true)]
public struct ConcatStr : IBinarySerialize
{
SqlString _concatenatedStrs;
bool _isEmpty;

public void Init()
{
_concatenatedStrs = null;
_isEmpty = true;
}

public void Accumulate(SqlString value)
{
if (value != SqlString.Null)
{
_concatenatedStrs += value;
if (_isEmpty==true)
_isEmpty = false;
}
}

public void Merge(ConcatStr group)
{
_concatenatedStrs += group.Terminate();
}

public SqlString Terminate()
{
if (!_isEmpty)
return _concatenatedStrs;
else
return SqlString.Null;
}
#region IBinarySerialize Members

void IBinarySerialize.Read(System.IO.BinaryReader r)
{
_isEmpty = r.ReadBoolean();
if (!_isEmpty)
_concatenatedStrs = new SqlString(r.ReadString());
}

void IBinarySerialize.Write(System.IO.BinaryWriter w)
{
w.Write(_isEmpty);
if (!_isEmpty)
{
if (_concatenatedStrs != SqlString.Null)
w.Write(_concatenatedStrs.Value);
}
}

#endregion
}

You shouldn’t forget to serialize the flag. Failure to do so wouldn’t preserve its value and it would always have the value in the initialization. If you put a breakpoint, in the terminate event, you could clearly say this one works.


image

And the result is what you expected:
image

The same workaround can be applied to any data type of your return value, be it a value-type or UDT. For example, if you don’t apply a workaround like the one below to an integer UAG, you get zero instead of null for an empty table!

[Serializable]
[Microsoft.SqlServer.Server.SqlUserDefinedAggregate(Format.Native
,IsNullIfEmpty=true)]
public struct SumInts
{
int _accumulator;
bool _isEmpty;
public void Init()
{
_isEmpty = true;
}

public void Accumulate(SqlInt32 value)
{
if (value != SqlInt32.Null)
{
_accumulator += value.Value;
if (_isEmpty)
_isEmpty = false;
}
}

public void Merge(SumInts group)
{
_accumulator += group.Terminate().Value;
}

public SqlInt32 Terminate()
{
if (!_isEmpty)
return new SqlInt32(_accumulator);
else
return SqlInt32.Null;
}
}

I still hope I would be proven wrong with my findings. But until then, I just have to content with this dirty codes. What’s important is that it works.

Sunday, February 8, 2009

IEqualityComparer.Equals() Not Called By Some LINQ Extensions

While I was reviewing the utility program I discussed in my last post, the geek in me suddenly kicked in. It’s trying to find other means of achieving what that program has already achieved. After a few minutes of rummaging the good ole’ Intellisense, it finally found them: Intersect, Except and Union. What I thought was a straightforward refactor turned out to be a one-hour ordeal. Along the way, I learned a fact about some LINQ extensions and a bad practice I’ve been so oblivious after all these years.

The extension methods Intersect, Except and Union are set-based operations that can be used in lieu of subqueries. They might be a little academic but using them makes the queries less complex and easy to read. One notable advantage they have over subqueries is that don’t need lamba’s as you see here:image

But I was surprised by the results. The one at the left was what I was expecting which was the result of the implementation in my last post.

image image

It was obvious that queries involving IEqualityComparer failed. Queries for update versions, new files and files to copy - although not the result I expected - were correct based on the operands used. My investigation zeroed in to nothing else but my implementation of IEqualityComparer. The question is: why are my IEqualityComparers not working in set-based operators?

Just like any coder, the first thing I did was create breakpoints in the Equals logic as shown here:

image

The result of the first run took me aback as much as the result of the query. The Equals methods were never called at all! Could this be a bug? Could it be that those methods are ignoring IEqualityComparer? I ponder on reporting this to MS as a bug but something was telling me I haven’t exhausted all possible causes. That something was that fellow in the corner and his name is GetHashCode().

Shifting the breakpoints to GetHashCode brought me to the next step. They’re indeed the ones called instead of Equals but that still didn’t help me figure out why the logic was failing. Well, I would be damned. It’s failing because I was using the wrong object to generate the hash code. It should be the property used in Equals instead of the object that defines the generic class. Fixing the lines as shown below finally brought end to this ordeal

image

image

So what?

Now I know that some extension methods have a special preference on GetHashCode. GetHashCode in turn gained a new level of respect from me. Call me stupid but I didn’t give a hoot about GetHashCode then. As far as I could remember, the only time I did was when I dissected CSLA. Good thing this glitch happened only to a pathetic program. An hour debug session on production codes is just too much for a simple omission like this.