NOODB: Serializer

May 20, 2010

For a while, I’ve been saying that I need to eliminate the custom serializer. That requires some elaboration.

The NOODB serializer converts an object graph to an xml string. The point of that was to allow it to send the string to a Wcf endpoint without the server having to be aware of the types. It eliminates all kinds of TYPE issues. If a property is defined as an interface, it tracks the actual implementation so it knows what to create.

That all works fine. But, later I realized it was unnecessary. Instead of converting it to xml, I can convert it to a graph of objects that are known by the client and the server and can be marshaled. The objects are, essentially, glorified name value pairs.

Because the client and server know these objects, we can pass them to a Wcf endpoint no problem.

But, the new “thing” would convert an object graph to something else for the purposes of persistence. So, it’s still a serializer.

Anyway, I made good progress on projections. But, I got to the point where I had to integrate it with some existing code, which resulted in mingling with some serializer stuff. Since I knew all of the serializer stuff was gong to change, it felt like a waste of time doing the integration work.

So, I’m in the process of eliminating the xml serializer in favor for my simple object graph. So far, it’s working for:

  • Primitives
  • Nullable Primitives
  • Complex Objects
  • Arrays

I still have 3 or 4 other types to do. This new serializer is MUCH cleaner; I’ve learned from the previous attempts. I expect the remaining types to go in pretty easily.

But, there’s more to it than that. Once done, I have to gut part of the server to get it in there.

So, it’s going to be a while. But, it’s an important improvement. I’ll keep you posted.

        public void Primitives()
            Guid guid = Guid.NewGuid();
            Guid guid2 = Guid.NewGuid();
            ObjectConverterTest test = new ObjectConverterTest
                Boolean1 = true,
                Boolean2 = false,
                Guid = guid,
                Integer = 993344,
                IntegerNull1 = null,
                IntegerNull2 = 3333,
                String = "test",
                SomeEnum = SomeEnum.Two,
                BoolNull1 = null,
                BoolNull2 = true,
                IntegerArray = new int[] { 1, 2, 3, 4, 5 },
                Nested = new ObjectConverterTest
                    Boolean1 = false,
                    Boolean2 = true,
                    Guid = guid2,
                    Integer = 2222,
                    String = "nested string",
                    SomeEnum = SomeEnum.One | SomeEnum.Three

            ObjectConverter converter = new ObjectConverter();
            Part serialized = converter.Serialize(test, null);

            ObjectConverterTest deserialized = (ObjectConverterTest)converter.Deserialize(serialized);
            Assert.AreEqual(guid, deserialized.Guid);
            Assert.AreEqual(993344, deserialized.Integer);
            Assert.AreEqual("test", deserialized.String);
            Assert.AreEqual(SomeEnum.Two, deserialized.SomeEnum);
            Assert.AreEqual(3333, deserialized.IntegerNull2);
            Assert.AreEqual(5, deserialized.IntegerArray.Length);
            for (int i = 0; i < 5; i++)
                Assert.AreEqual(i + 1, deserialized.IntegerArray[i]);

            Assert.AreEqual(guid2, deserialized.Nested.Guid);
            Assert.AreEqual(2222, deserialized.Nested.Integer);
            Assert.AreEqual("nested string", deserialized.Nested.String);
            Assert.AreEqual(SomeEnum.One | SomeEnum.Three, deserialized.Nested.SomeEnum);

NOODB: Projections in Progress

May 17, 2010

When you get all of the reviews for an author, it takes a while to load because of the review text; it’s reviewing a lot of data it doesn’t need. So, the needs of DvdFriend dictate that projections are next on the list.

To complete projections, I’m going to have to remove the custom serializer. That’ll take some work. But, my immediate need is to exclude a field.

from r in MovieReview select r.**, –r.ReviewText where r.Author.ScreenName=’DvdFriend’

r.** means to do a DEEP GET; it will get the entire graph unless a subsequent instruction tells it otherwise.

-r.ReviewText instructs it to remove that field from the select list. When the object comes back, the ReviewText property will be null.

The SELECT statement is fully parsing. I made a few changes in the build object to support this. There are two things left:

  1. Per object type in the graph, figure out which fields are needed. The server will do this work and pass the result to the sql object
  2. Get the select from the resolver (previous bullet) to the build object code. I’ll have to massage some things for that.

I’ll work on that this week as I can. Hopefully I can have it done in a couple days, but we’ll see.

In other news: I had some time to kill today, but not enough time to get the gears turning for real work, so started making the code style cop compliant. I have to do this every few weeks; I don’t keep up with as I code.

NOODB: Property Drilling Works!

May 14, 2010

I got the property drilling working last night, but it was ugly and only worked under specific conditions. I had to take a step back and review all of the moving parts. It needed to be slapped around.

The logic of the QueryParser is motly unchanged, but the objects it populates have changed. The output is stored differently.

The SqlQueryConverter has been seriously overhauled.

The result is a much cleaner API. The output of the QueryParser needs to be easily consumable so that it can be converted to run against any repository. I’m very attentive to the public facing APIs. I write the SqlQueryConverter as if I know nothing about the innards of NOODB.

The final result is pretty cool. As I’ve been describing for days, I can now drill into the object graph for the where and the order by. This will later encompass the select as well.

Once everything was working, I went back to what started this to begin with. The DvdFriend page shows a list of recent reviews. You can click the AUTHOR link to see all of the reviews that author has written.

  1. Created the UserReview page
  2. Created an object data source that hits a new method (below)
  3. Created a grid view that binds to the object data source
  4. Tweaked the grid
  5. Updated the recent reviews to link to this new page
    public static Collection<MovieReview> GetUserMovieReviews(string userName)
        IObjectClient client = NoodbUtility.GetNoodbClient();
        return client.Query<MovieReview>("from r in MovieReview where r.Author.ScreenName=='" + userName + "' orderby r.ReviewDate descending");

That’s a pretty simple query. It just returns the movie reviews of a particular author.

The one I tested with is a little more complex.

@"from m in MovieReview 
                        m.Author.ScreenName == 'DvdFriend' 
                        || m.Author.ScreenName=='Chris' 
                        m.Author.ScreenName ascending, 
                        m.Movie.ProductName descending 
                  take 10"

I didn’t wimp out on the sql; it does the bare minimum number of joins to get what it needs. In this case, it needs Author.ScreenName 3 times, but the sql only gets it once. The easier/quicker way would’ve been to join to it 3 times, but then I’d feel just awful about myself.

The user review page was up and running in about 4 minutes, most of which was spent tweaking the grid.

  • Added Movie Title
  • Removed Review Id
  • Removed Review Text

Here’s a good example of where the view feature and/or projections will come in handy. First of all, the full review comes back in the object. I’m not displaying it, but it’s still coming back. It would be nice to exclude that.

Secondly, the grid is only showing 5 fields. Not only should we be able to specify that we only want those 5 fields, but we should also have the option to bring them back in a flat format rather than the full object hierarchy. That’s great in read-only scenarios.

If NOODB had a road map, tonight would be a significant benchmark. This is a really powerful feature.


NOODB: Property Drilling

May 12, 2010

I finally started working on it tonight.

The parsing is complete, as are the changes to the API to store all of the nested information.

I started to work on the SQL, but it’s late and I’m tired. That’ll have to wait for another day. Unfortunately, I can’t really use any of the existing SQL generation code. The existing code is based on an object type; it builds a CTE that reveals all of the properties for the object type.

The new one will be more concise, and will span multiple objects. It will only retrieve the properties that it needs to execute the query. (Later, this will be combined with any fields specified in a projection).

This is the query I’m working on

from m in MovieReview where m.Author.ScreenName=’DvdFriend’

I don’t have a test for it yet, but you can drill as far as you want. IE:

from m in MovieReview where m.Something.SomethingElse.Something.Author.ScreenName=’DvdFriend’

In conclusion, it’s going to be brutal. But, it’ll be worth it. I’m going to need some Wheaties.

NOODB: Property Drilling, your time has come!

May 10, 2010

It’s time to do the next big thing. I thought it was going to be projections, but alas, it wasn’t meant to be.

The Recent Reviews page shows the author of the review as a link. When you click that link, it’s supposed to show you all of the reviews from that author. I thought I’d knock out that page real quick, but NOODB doesn’t support what I need it to do. Whoops. Now I know what’s next. (Well, strictly speaking I COULD do this by pulling back all of the records and filtering them on the client… but that’s brutal and wouldn’t set a good example for all of those people that don’t read my blog.)

I call this “property drilling” because I am a simple minded fool. I need to read the MovieReview objects and drill down into the author and look at the user name. I need to be able to do this.

from r in MovieReview where r.Author.ScreenName = ‘DvdFriend’

NOODB supports can do r.x but not r.x.y or r.x.y.z; at least not yet.

This will work with collections too. (IE: if any element of the collection matches the criteria, take it). The SQL will have to span one more table, and I’ll have to add a distinct, but it won’t be much more than the non-collection version.

Easy to type, but there’s a lot to implement. I lost a lot of sleep last night thinking about how I would represent that in the query object. The SQL shouldn’t be much of a big deal; the one thing I’ll have to deal with is the possibility of duplicate column names when building the CTE across multiple objects.

Here it is, 10:27pm. On one hand, I’d like to get started on this, but then I’ll probably be up until 2. I think I’ll go play Splinter Cell instead, and leave this for tomorrow night. (That’s a fine idea, as long as I don’t lose sleep mentally coding it again).

NOODB: Order By

May 9, 2010

Yesterday, I hacked the order by just to see it work.

Today, it works for real. This is a method that the RECENT REVIEWS page uses to get the reviews. (Repeater => Object Data Source => this method)

    public static Collection<MovieReview> GetRecentMovieReviews(int count)
        IObjectClient client = NoodbUtility.GetNoodbClient();
        return client.Query<MovieReview>("from r in MovieReview orderby r.ReviewDate descending take " + count);

As usual, kindly overlook the fact that there isn’t a select. I don’t really need it until it supports projections, which is coming soon.

noodb orderby

That’s a screenshot of the website. None of the recent movies have images. If they did, you’d see them on the left side. (There is one off screen; Roswell Season 1)

Next, it would be nice to query for a single field so that I can retrieve the FULL REVIEW through ajax. I think that’ll be my next move, but I’m not committed yet. That’d work like

from r in MovieReview where r.ReviewId=’…’ select r.ReviewText


May 8, 2010

TAKE is the LINQish equivalent of top. But, it doesn’t exist in the query expression; it’s an extension method.

I’m just tacking it on to the end of the query, at least for now.

I added a RECENT REVIEWS page to the DvdFriend test site. The NOODB query is:

from r in MovieReview take 20

Of course, there should be a SELECT in there, but I’ve been avoiding that.

The next thing I need is to “order by ReviewDate descending”. I haven’t done that yet, but did manually tack it onto the end of the SQL query just to see it work. It worked fine once I addressed the order issues covered in the previous post.

I looked at the linq syntax for order by. When you do it via code, the first sort field is an OrderBy, and all subsequent sort fields are specified by ThenBy. But, looking at the query expresions, I only see ORDER BY. Apparently, it does the THEN by automatically. I’ll find out soon enough.