NOODB: JQuery Playing Continues

June 24, 2010

I changed my page to use the jquery dialog rather than the manual thing I was doing before. It’s not working 100% the way I want it to, but I’m trying not to get hung up on the details.

I created a WCF restful service for the page to call so that it can run queries. This service is temporary; the noodb server will expose the services directly at some point. For now, I just have a simple proxy into it.

The JQuery stuff is neat, and the documentation is helpful. Implementing the ajax calls was a straight forward process. The error handling on it isn’t great; If the server blows up, jquery doesn’t let you know. You can attach an event to the XmlHttp object, but it only tells you that there was a failure. The failure isn’t associated to a particular ajax request. In this matter, ASP.net ajax works better. You pass it two methods: the successful callback method, and the failure callback method. It works much better than the jquery approach.

noodb admin page

This doesn’t yet show nested objects. The information comes back just fine, but it’s not rendering.

Advertisements

NOODB: Not much of an update

June 20, 2010

I’ve been taking it easy the last few days. Finishing projections was a big deal. There are about 1,000 more things to do, but the fact that I can build 90% of DvdFriend.us without any additional NOODB means that I can take a breather.

As I build the NOODB demo version of the DvdFriend site, I want each page to display the queries that were executed to construct the page. I goofed around with that a bit, but wasn’t really in the mood.

I also want a NOODB admin page so you can go run some queries and see how it works. I hate building web pages. I spent some time learning some basic JQuery things so that I can use it. The result is a QUERY window and a QUERY RESULTS window. Both are draggable and resizable. I’m having zindex trouble, but otherwise it looks ok.

I have a plain text banner on the top that displays NOODB, and the tentative slogan of  “For when you just want to save some objects”.

I leeched part of an image to put behind the window title bars. It’s too short, so doesn’t look right. I’m not excited about that.

Lastly, I put a picture taken by the Hubble telescope as the background. Swell.

The dilemma is that I need decent looking pages and a decent looking admin site so that cosmetics atrocity don’t turn people away without running some queries. The problem is that I really have no interest in building any web pages. It’d be swell to have some help on those aspects of it.

Oh well, I’ll keep picking away at it. When I get too bored, I’ll switch to something more interesting. As I said, there are still plenty of things to do.

Screenshots coming soon.


NOODB: Projections Complete?

June 13, 2010

Last night’s post included some samples showing that the property resolution was working. One of them actually wasn’t due to a recursion problem. That was a minor fix.

After that, plugging everything in went pretty easy. I thought that I might have to change some things around due to how the property resolution ended up working, but I didn’t. Everything fit together well.

I ran some tests to see how performance is under a variety of scenarios. Remember that I haven’t paid any particular attention to performance; I’m keeping it mind as I go, but I’m not testing and analyzing yet. These are the first-pass numbers.

Each test is run 10 times. Additionally, the server is warmed up before the first test executes.

All tests retrieve every movie review from the dvdfriend database.

Get Everything

This simply calls client.GetObjects<MovieReview>(). This is just a read of every object of the given type. It’s the full graph. You probably don’t want to use this in most situations. It’s like doing “select * from table”.

GetEverything
Iteration 0: 2006.4515ms, Count=2904
Iteration 1: 1841.9566ms, Count=2904
Iteration 2: 1830.4992ms, Count=2904
Iteration 3: 1806.5747ms, Count=2904
Iteration 4: 1894.1823ms, Count=2904
Iteration 5: 1806.3163ms, Count=2904
Iteration 6: 1732.0549ms, Count=2904
Iteration 7: 1692.3944ms, Count=2904
Iteration 8: 1788.4181ms, Count=2904
Iteration 9: 1769.857ms, Count=2904
Average: 1816.8705

Query Everything – No Select Clause

This runs a query without specifying a select clause. Because there isn’t a select clause, it will automatically retrieve the entire graph.

QueryEverything_NoSelectClause
from r in MovieReview
Iteration 0: 1862.5085ms, Count=2904
Iteration 1: 1768.9057ms, Count=2904
Iteration 2: 1805.549ms, Count=2904
Iteration 3: 1810.9975ms, Count=2904
Iteration 4: 1809.6719ms, Count=2904
Iteration 5: 1747.9265ms, Count=2904
Iteration 6: 1814.4942ms, Count=2904
Iteration 7: 1801.1459ms, Count=2904
Iteration 8: 1789.6214ms, Count=2904
Iteration 9: 1722.5566ms, Count=2904
Average: 1793.33772

Query Everything – Deep Select Clause

The end result of this is the same as “Query Everyting – No Select Clause”. The difference here is that we specify r.**, which tells it to get all of the fields.

So what’s the difference? The difference is that when you don’t specify the select clause, it doesn’t resolve the properties, it just gets everything. With the select clause of r.**, it resolves all of the properties. Every query receives a list of all of the properties.

An optimization here would be to detect if all fields are explicitly selected. If so, then drop the projections. But, the hit isn’t that big, so I’m not in a big rush for it. (I will put it on the to-do list).

QueryEverything_DeepSelectClause
from r in MovieReview select r.**
Iteration 0: 1854.0705ms, Count=2904
Iteration 1: 1868.8928ms, Count=2904
Iteration 2: 1989.5221ms, Count=2904
Iteration 3: 1849.2452ms, Count=2904
Iteration 4: 1799.233ms, Count=2904
Iteration 5: 1823.9078ms, Count=2904
Iteration 6: 1817.7604ms, Count=2904
Iteration 7: 1846.124ms, Count=2904
Iteration 8: 1848.9052ms, Count=2904
Iteration 9: 1789.4011ms, Count=2904
Average: 1848.70621

Query Everything – Shallow Select Clause

Now we start to see some real benefits. Rather than get the entire review graph, it only gets the top level properties.

QueryEverything_ShallowSelectClause
from r in MovieReview select r.*
Iteration 0: 814.5349ms, Count=2904
Iteration 1: 848.7935ms, Count=2904
Iteration 2: 855.4855ms, Count=2904
Iteration 3: 842.2453ms, Count=2904
Iteration 4: 823.8852ms, Count=2904
Iteration 5: 793.6476ms, Count=2904
Iteration 6: 743.9888ms, Count=2904
Iteration 7: 789.7292ms, Count=2904
Iteration 8: 897.9049ms, Count=2904
Iteration 9: 871.061ms, Count=2904
Average: 828.12759

User Reviews – Without Projections

This is the use case that prompted the implementation of the projections. Obviously they were always a required feature, but I didn’t need it until building the User Review page in DvdFriend. The user review pulls back all of the user’s reviews, including the review text which isn’t shown. It only needs 4 fields; it’s pulling back everything else for no reason.

UserReviews_WithoutProjections
from r in MovieReview where r.Author.ScreenName=='DvdFriend' orderby r.ReviewDate descending
Iteration 0: 1012.1361ms, Count=1293
Iteration 1: 948.4727ms, Count=1293
Iteration 2: 927.2112ms, Count=1293
Iteration 3: 923.0019ms, Count=1293
Iteration 4: 844.9469ms, Count=1293
Iteration 5: 924.5507ms, Count=1293
Iteration 6: 985.8055ms, Count=1293
Iteration 7: 927.8918ms, Count=1293
Iteration 8: 913.6041ms, Count=1293
Iteration 9: 903.036ms, Count=1293
Average: 931.06569

User Reviews – With Projections

Ta da! Now we only pull back the exact fields that the user review page needs.

UserReviews_WithProjections
from r in MovieReview where r.Author.ScreenName=='DvdFriend' select r.ReviewDate, r.Movie.ProductName, r.Rating orderby r.ReviewDate descending
Iteration 0: 614.1987ms, Count=1293
Iteration 1: 589.8489ms, Count=1293
Iteration 2: 552.8296ms, Count=1293
Iteration 3: 540.0005ms, Count=1293
Iteration 4: 552.5914ms, Count=1293
Iteration 5: 554.0354ms, Count=1293
Iteration 6: 543.7779ms, Count=1293
Iteration 7: 552.0717ms, Count=1293
Iteration 8: 585.1147ms, Count=1293
Iteration 9: 596.9977ms, Count=1293
Average: 568.14665


NOODB: Projection Update

June 12, 2010

When I last wrote, I was planning on working on the property resolver. That didn’t happen. Sleep happened.

This was a busy week. I was in training for the week, so there was a lot of driving. I didn’t have any time left to work on this, but I’ve been thinking about it as time permits.

I was pondering the most elegant and efficient way of dealing with the projections. Every time I thought I had it, I’d keep thinking about it, and come up with a scenario where it wouldn’t work.

Tonight, I was finally able to sit down and work on it while watching SGU, and a variety of other shows. The fields aren’t being passed to the data object yet, but everything is being properly parsed and it knows what needs to be retrieved.

The following is output of test code. The test code runs the parser against a query, then converts the select output to text.

As previously described, .** means select everything from the current object and all of it’s children. .* means just select the simple properties from the current object. The rules are applied left to right.

Notice that when you include a property, it includes all of its ascendant properties. That’s so there aren’t any gaps in the graph as we retrieve and reconstruct the various parts.

from p in QpBlah select p.**, -p.Author.Address.**, p.Author.Address.State.Abbreviation
include: Root2
include: Author
     include: Name
     include: Address
          include: State
               include: Abbreviation
               exclude: Name
          exclude: Street
          exclude: City
include: Root1


from p in QpBlah select p.Author.Address.State.Abbreviation
exclude: Root2
include: Author
     exclude: Name
     include: Address
          include: State
               include: Abbreviation
               exclude: Name
          exclude: Street
          exclude: City
exclude: Root1


from p in QpBlah select p.Root1
exclude: Root2
exclude: Author
     exclude: Name
     exclude: Address
          exclude: State
               exclude: Abbreviation
               exclude: Name
          exclude: Street
          exclude: City
include: Root1


from p in QpBlah select p.Author.Address.State.**
exclude: Root2
include: Author
     exclude: Name
     include: Address
          include: State
               include: Abbreviation
               include: Name
          exclude: Street
          exclude: City
exclude: Root1


from p in QpBlah select p.Author.Address.State.Abbreviation
exclude: Root2
include: Author
     exclude: Name
     include: Address
          include: State
               include: Abbreviation
               exclude: Name
          exclude: Street
          exclude: City
exclude: Root1


from p in QpBlah select p.Author.Address.State.**, -p.author.address.state.abbreviation
exclude: Root2
exclude: Author
     exclude: Name
     exclude: Address
          include: State
               exclude: Abbreviation
               include: Name
          exclude: Street
          exclude: City
exclude: Root1


from p in QpBlah select p.Author.**, -p.author.address.state.**, p.Root1, -p.author.address.**
exclude: Root2
include: Author
     include: Name
     exclude: Address
          exclude: State
               exclude: Abbreviation
               exclude: Name
          exclude: Street
          exclude: City
include: Root1


from p in QpBlah select p.**, -p.author.address.**, p.author.address.state.**
include: Root2
include: Author
     include: Name
     include: Address
          include: State
               include: Abbreviation
               include: Name
          exclude: Street
          exclude: City
include: Root1



In conclusion, this is a big piece, and I’m glad it’s done. Next, I have to plug it into the server and the db parts. I already set that all up, but may have to change it since the resolver has evolved so much. Maybe tomorrow.

Getting close…


NOODB: It’s alive (again)!

June 6, 2010

This post is a couple days over due.

When last we left NOODB, I was carving out the custom xml serializer which ended up not being necessary. Whoops. Instead, it “serializes” into an object graph that is, essentially, nested name value pairs. When the server is local, it can pass a reference. When the server is remote, the object can be passed via WCF. There won’t be any known type issues.

I’ve been saying that needs to be done for a while. It became necessary when I started working on projections. (In particular, I need to be able to do  “from r in MovieReview select r.*, –r.ReviewText”, or something like that.) That involves analysis of the final set of selected properties and passing it to the data context. It’s the last part that prompted the gutting; I would’ve had to put it in a lot of places that I knew was going to be cut out, so didn’t want to do it twice.

That’s all done. NOODB is back to 100% functionality without the custom serializer. Additionally, it passes the property ids to the data context.

The next big thing, which I’ve been stalling on for the last couple days, is the property resolution part. Based on the fields and patterns you select, it has to figure out all of the property ids. It’s a daunting task that’s hard to commit to while coding and watching tv at the same time. This is one of those things where I should go up to the office and focus on it, but we just had a baby. I don’t drag my butt upstairs any more than I need to. Besides, I need my TV.

All of the meta data for the analysis is collected and ready to go. The big piece is resolving it. It’s complicated. Consider, for example that you have an object A with a property B, and B has a property C. The select may say “a.Property1, c.Property3.”

There are 3 objects and we’re not selecting anything from the one in the middle. As it figures out the properties to get, it has to know that “hey, even though they didn’t request anything from b, I have to get b anyway in order to get to c.”

Logically, does this make sense? B will come back empty except for the one property that is C. No, it probably won’t make sense, but that’s not my decision, it’s yours. The syntax of the query allows you to specify stuff like that, so the query engine will do stuff like that. Whether or not it’s a good idea is up to you. If your just dumping a bunch of stuff to a grid or some other read-only presentation, then it may be just fine.

In a related story, I’ve put a lot of thought into the syntax of the selects, and here’s what I’m going with, at least for now.

It’s going to start with the first projection, then just keep applying the other projections on top of it. The last ones win.

r.*, –r.ReviewText

Get all everything in R except for the review text. r.* is shallow; it won’t get child objects, just the properties of R.

r.**, –r.Movie.Director, –r.Author, r.Author.Name

Get everything in R, including the child objects. Then remove the movie’s director, and the review’s author, then put back the author’s name.

Once this is all working, the next step will be to return the values in a flat format rather than a graph. If you just want to dump the results onto a grid or something, then you don’t need the graph, you just want some values.

Tonight, I begin work on the property resolver. Tomorrow, I rest.


NOODB: Baby Delay

June 1, 2010

Greetings

I haven’t worked on NOODB in several days due to the birth of Ellie Rose Allard: 7 pounds, 3 ounces, 19 inches long.

As you can imagine, I’ve been very busy not sleeping.

Things are starting to settle down a bit, and I hope to resume coding some time this week. Watch for the 1am updates.