Application Server – Pubsub Workign

April 7, 2013

I haven’t really worked on my latest personal project in a few weeks. The application server is up and running, and is completely command driven, and the commands can be specified declaratively. 

I started to implement the new SEMANTIC LOGGING BLOCK from P&P, and created a custom sync that sent the log items to MSMQ. That worked, but I want to be able to process the messages multiple times. IE: put them in a database, and also monitoring tools and the website. So, I needed pubsub.

Since that realization, I haven’t worked on it too much. I’ve spent a few minutes here and there, but tonight was the first time I put some serious time into it while watching “MASK OF ZORRO”.

The plan is to have 2 implementations out-of-the-box

- MSMQ – the best choice, when available

- NETTCP – a less reliable backup for when MSMQ isn’t available.

The nice thing is that both implementations can use the same contracts. The only thing that will change is the bindings. Yay for WCF! But, I have been focusing on MSMQ.

As of tonight, it’s working, but there’s still a lot more to do. The client is pretty configurable, but the server isn’t yet. There’s a few hard coded things in there that have to go. All of this can be driven by the WCF configuration files, and that will be an option, but I want it to just work without people having to mess with anything. I have some work to do on that end.

There are two WCF services running on the app server: 1 to handle subscribes, and 1 to handle publishes. A lot of the work I did previously payed off nicely here. I created my WCF contracts and a MSMQPubsubFeature. I put two attributes on the feature, one per service, and now the services automatically start… just as they were supposed to. (It defaults to NETTCP, so I added WCF configuration to the app.config to switch it to MSMQ.)

The publish service:

[WcfService("MSMQServerPublish")]
public class PublishService : IServerPublishService

The subscribe service:

[WcfService("MSMQServerSubscribe")]
public class SubscribeService : IServerSubscribeService

The feature class:

[Name("MsmqPubSubFeature")]
[HostWcfService("MSMQServerPublish")]
[HostWcfService("MSMQServerSubscribe")]
public class MsmqPubSubFeature : Feature, IPubSubFeature

Each service has a name. The feature has a name. When the feature starts, it starts the two WCF services by name.

I setup a simple test application that uses the pubsubclient (a stand-alone assembly) to exercise it, and it’s working as expected.

Progress made.

 

 


Something I wrote in 1996

April 3, 2013

I’ve been trying to clean up and organize some old files and archives. I wrote this in March 1996 when DVD was being introduced.

As I started reading it, I immediately groaned. These days, I would never presume anything to be “entertaining” or “fine”, I’d just write it and let readers judge from themselves. Oh well.

From the Dawn of Time,

To D.V.D…

The Evolution of Mass Storage

An Entertaining Documentary written by the  fine people of

 Allard-Works Enterprises.

Note:This is written to be somewhat entertaining, but the technical content is accurate.

D.V.D. is an acronym of varying meaning.  To some, it means Digital Versatile Disk.  To others it means Digital Video Disk.  And to a few, it sounds like a brand of quality underwear.  Whether the V actually stands for Video or Versatile depends on where you read it.  However, because D.V.D. does so much more than just video, Versatile is much more appropriate.  As for the underwear people… don’t worry about them.

D.V.D. is the latest entry in the evolution of mass storage.  As are many things when they are first created, it is a phenomenal technological breakthrough that will soon have a significant impact on the multi-media world.  But before you, the reader, can fully appreciate D.V.D. and all it’s splendors, a quick refresher course of computer history is in order.

In the beginning, there was nothing.  The universe was an infinite void, much like the interior of Rush Limbaugh’s head.  And then there was a magnificent explosion of matter, tossing planets and galaxies every which way, filling the infinite void with a great number of floating rocks.  After a few million years, one of those rocks metamorphosed into a prettier rock, which was shortly thereafter named the planet Earth.  The Earth’s molten-lava surface cooled down allowing forests to rise up from the ashes, and oxygen and hydrogen combined to create a substance that we now use to wash our cars.  Though there was no one around to realize it, it was in those early years of the Earth’s childhood that the first form of mass-storage was created.  Those same forms of mass-storage are still around today.  We call them lakes and oceans.  When you think about it,  we have to keep the water somewhere!

For a long time, the Earth revolved around Ra, not really doing much other than chatting with the moon.  Meanwhile, on the surface, activity was afoot.  After a few failed attempts at creating significant life, a few amino acids and proteins finally got together, had a party, conjugated, and created the original Jurassic Park.  Dinosaurs ruled the planet for more than 6 months, and then another one of the universe’s rocks killed them.

After that, computer history varies depending on your religion (or lack thereof), but, one way or another, homo-sapiens began their reign.  Humans all across the world soon began to communicate with one another, form societies, discuss problems, and kill each other.  After many years of this, an inspirational man with a powerful vision came along to teach his brothers and sisters of the world.  This man believed himself to be the son of God, the almighty messiah, and many worship him to this day.   Over the years he has been called many things by many people… but most of us just call him Bill Gates.

Humans, it is believed, first created computers on this planet.  Now, let’s explore why.

Data, otherwise referred to information, is the basis of nearly every activity in existence, ranging all the way from going to the bathroom, to controlling a space shuttle.  At about the same time humans overcame their fascination with fire, they began to realize the significance of data, and the importance of storing it for future reference.  This lead to the creation of writing which is, essentially, data storage.  People have been writing for at least 2000 years, and data has been accumulating for the entire duration.  Humans are fascinating creatures in the aspect of their overall desire to learn, creating more data based on old data, and constantly progressing.  Industries were born and flourished, and the more data we created, the more we needed, and wanted.

This whirlwind of data eventually led to the discovery of electricity, Pop Tarts, and atomic structure, which later led to the computer.  A computer, in simple terms, is a device that stores, manipulates, and utilizes data to produce different data that the user desires.  The difference between the computer and humans is the speed in which they do this.  Also, humans can’t play Doom with just a pad of paper.

To break it down even further, computers work with electrons, which as most Physic majors will attest, are very very small.  For each of the billions of instructions the computer processes, each one breaks down into a simple term: YES or NO.  Yes is represented by the presence of a few electrons in a specific area, and NO is represented by the lack of electrons.  Because electrons are so small, millions of these yes and no questions answers can be stored in a very small area.  As computer technology progresses, the number of electrons necessary to indicate a YES decreases.  Someday, in the near future, only one electron will be needed to perform this task.

Each of these YES and NO results are called a bit.  Eight of these bits form a byte, which is usually the smallest piece of data the average user uses.  For example, each character you read on this paper is represented by one byte, which consists of 8 yes and no’s.

The problem with computers is that they require a constant feed of electricity to maintain the integrity of it’s bytes.  Thus, when the computer loses power, it loses it’s memory.  Early on someone realized that this would be very bad in most cases, and technology developed ways to store the computer’s data.  As far as the personal computer is concerned, this led to the invention of the floppy disk in the early 80’s.  The first consumer friendly floppy disk could hold 360k (360,000) bytes of data on a 5.25 inch disk.  That is a massive amount of data.  Time progressed, and being as greedy as we are, we wanted more storage space.  In August 1984, a company called IBM introduced a 1.2M drive.  (1 million, 2 hundred thousand. pronounced “1.2 meg drive”).  This disk stored four times as much as it’s predecessor in the same amount of physical area.  A 400% improvement.

At the time, 1.2M was incredible, and it is likely people never thought that they would need any more.

In 1986, the next floppy disk was introduced: The 720k.  720k, representing 720 thousand bytes, may seem a downgrade from previous successes.  However, the significant thing about the 720k is the fact that it is a much smaller, much more durable disk.  In relation, twice as much data as was held on the original 360k floppy could then be stored on a smaller disk.  In 1987, the 720k became the 1.44m drive.  Nine years later, the 1.44meg drive is still standard.

As floppy drives developed, so did another handy-dandy mechanism called the hard drive, or, fixed disk.  A floppy is just a disk that you can insert, use, and remove.  A hard disk, on the other hand, remains inside your computer.  It is much faster and holds a lot more information than a mere floppy.  The earliest hard drives could hold 10 megs of data… that’s 10 million bytes, or 80 million bits.  Again, everyone thought this was fantastic.  In the July 1989 Computer Shopper, 10 megs was considered mass-storage.  However, as the storage capacities increased, so did the amount of space needed for computer software.  Thus, between 1989 and 1996, hard drive sizes have risen from 10 megs, to in excess of 1,600 megs (1.6 Gigs), and are continuously growing in size while depreciating in cost.

Currently, CD-ROMS (Read only memory) are extremely popular.  The great things about CD-ROM is that they hold 650 megs of data on a 4.72 inch diameter disk which is only 1.2mm thick.  (Incidentally, it is reported that this size was decided upon because it is precisely large enough to hold Beethoven’s Ninth Symphony.)  Six-Hundred and Fifty Megs!!!  Now we’re playing with power.  This kind of mass storage on such a small, lightweight, portable device has opened up an entire new world in the realms of computer graphics and audio.   In the days of floppies, one would have to insert many disks into the drive during a long and tedious installation process.  With CD-ROMS, you just insert one disk and let it rip.

Let’s see how a CD works.  CD’s store data by using PITS.  A pit is a microscopic depression in the disk.  Where there’s a pit, there’s a YES.  Where there isn’t, that’s a NO.  These pits are 0.12 microns deep, and 0.6 microns wide… about the size of Rush Limbaugh’s brain.  A laser system is used to scan the disk in specified areas to see if a pit resides there or not, and returns the result to the computer system.

How much information, exactly, can a CD hold?  Well, I already mention that it’s 650 megs of data.  To give you an idea in practical terms, one CD can hold 74 minutes of high-fidelity audio, or 333,000 pages of text, or a combination of the two.

Some of us have been exposed to “LASER DISKS” which are very large CD’s that you can also use as a bathroom mirror.  Laser disks work on the same principles as CD’s, except they’re primary purpose is video.  Over the last few years, more and more companies have been putting actual video onto the CD, incorporating it into presentations and games.  The term “interactive movie” is becoming increasingly popular in the “intertainment” world.  Actors and actresses are being hired in place of the artists who used to have to conjure up fictional characters for their games.

SIDEBAR:  This doesn’t really fit into the paper anywhere, but think about this:  Years ago, the computer industry and the movie industry were at opposite ends of the spectrum.  Movie companies would hire actors and actresses to participate in a story.  Software companies would hire authors to create dazzling graphic and game characters.  Today, the two are meeting in the middle of the spectrum, and starting to pass each other, each advancing towards where the other began.  As I said, interactive movies are becoming more and more popular.  Two recent MAJOR games, Wing Commander III and Wing Commander IV hired a sleu of big time actors to  bring a video game to life.  Malcom McDowell, Mark Hammill, and Ginger Allen among them.  Meanwhile, “CG” (computer graphics) in the movie industry are beginning to replace actors and actresses.  In the movie Mortal Kombat, for example, there was a fight scene between two characters, where actors weren’t used.  It was all CG.  Similarly, in CASPER, most of the characters were computer concoctions, along with some key scenes from Jurassic Park.  The mother of all CG movies so far has been TOY STORY.  100% Computer Graphics, and it was a huge success, inspiring such things as the upcoming Disney movie “JAMES AND THE GIANT PEACH”, which uses a great deal of CG.  Recently, the movie JUMANJI dazzled audiences with it’s own computer generated jungle wild life.  Hundreds of thousands of lines of computer code were written for the lion alone.

            This leads me to the final point of the sidebar.  So far, CG has created dinosaurs, monkeys, cartoon characters, lions, and a distant shot of humans.  How much longer do you think it will take until someone creates a life like CG Human?  There is a man in L.A. (name lost) currently creating a humungus program which is, essentially, a CG human.  He is accurately reproducing all muscle groups of the human body, and how they interact with each other.  When this is done, he will have a programmable human that will, on the screen, do anything he tells it to, and the body will react just as a human will react.  As of January 1996, the head and upper right part of the body were done, with only one problem: the shoulder.  Apparently (if I remember correctly), much like gravity, no on understands how the shoulder muscle group functions.

            Envision a day 10 years from now when movie companies won’t have to pay 20 million dollars for a big name star.  All they’ll have to do is define the parameters of the body, then tell it what to do.  What will happen to all the big time actors then?  They’ll probably be in video games.

 

For years, people have been trying to cram as much video as possible onto a CD-ROM.  Unfortunately, it hasn’t been very succesful.  A CD can hold about twelve seconds  (mostly a guess, I lost the actual figure)  minutes of uncompressed video and audio.  Since most movies are in excess of 90 minutes, something had to be done.  That something turned out to be MPEG (Motion Pictures Experts Group).  MPEG can compress at a ratio of 200 to 1 maintain high quality.

How does compression work?  Good question.  Just a brief answer:  Compression uses a few bytes to represent many bytes.  This is a loose example that I am making up on the spot, but let’s pretend that you have a document with the word THE in it 50 times.  That 150 characters, or, 150 bytes.  Using compression, every THE might be replaced with ~, bringing the bytes needed down to 50 characters instead of 150.  Then, every time the software sees ~, it replaces it with THE.  This is an extremely uncomplicated example, but the principle is accurate.

MPEG is much more complicated example of compression.  If you’re watching a movie, and someone is talking, what is changing about the video?  Not too much… just the speakers mouth, and maybe a little head movement.  MPEG pays attention to these changes.  Instead of storing an entire picture frame, it only stores the part that has changed from one frame to another.  So, the less movement in a film, the more tightly it can be compressed.  If you’re watching a basketball game where everything is panning and moving, then the compression isn’t as good.  High end home computers can do MPEG compression.  However, the second generation of MPEG is in full swing.  MPEG 2 produces even tighter compression, and it requires hardware to do the compressing.  MPEG 3 is soon coming.

Back to CD’s.  Even with all of this major compression, there still isn’t nearly enough room on a CD to fit an entire movie.  A lot of companies were considering breaking it down into two CD’s, but it never really happened.  Recently, however, music groups such as THE CRANBERRIES are incorporating videos on their CD’s in addition to the audio tracks.

The original concept for CD’s was concocted way back in 1983.  Thirteen years later, it is an extremely crucial part of most computer systems.   Thirteen years old… that’s ancient in the computer field.  It would stand to reason, as history dictates, that it’s time for a change.  That change is upon us, as you read in the opening paragraphs of this essay.  That change is D.V.D: Digital Versatile Disk, and it’s coming soon.

D.V.D. is the next step in the evolution, and it is a direct descendant of the CD.  Over the last few years, many companies have had different ideas on how to increase the storage space of the CD.  One idea was a two sided CD, with two lasers, and a disk twice as thick.  Another idea was a double layer CD, but only one laser.  The laser would just focus on the appropriate level, and ignore the other.  Another idea was to just make bigger CD’s with more, smaller pits.

The result is a dual layer disk of the same thickness (1.2mm), but a diameter that extended from 4.75 inches to 5 inches, and pits that have significantly decreased in size.  This, combined with the MPEG 2 Video compression standard results in a 4.7Gigabyte single layer disk, and a 9.4Gigabyte dual layer disk.  Single layer will provide for 133 minutes of full quality video along with three sound channels, and 4 subtitle channels.  The dual type disks will allow for 4 1/2 hours of the same.  As for speed, the slowest DVD player will be 8x to 10x faster than the original CD player, which is the speed that audio players use.  The fastest CD-ROM player of today is only 8x.  The jump from CD to DVD is monstrous, and the beautiful part is that it’s backwards compatible.  You’re low-storage 650 meg CD-ROMS will operate in the mass-storage 9.4 Gigabyte DVD drive.

D.V.D. will be making it’s debut in late 1996 or early 1997, and costs will begin at $500 to $700.  As more companies produce them, the price is sure to drop, and the speed is sure to go up.  Unfortunately, D.V.D. is still a WORM device (Write Once, Read Many) but, at last, entire movies will fit on one disk.

In 1983, a floppy drive could hold 360,000 bytes of data.  In 1996, D.V.D can hold 9,400,000,000 bytes.  Why will we ever need more space than that?  Well, as history shows, in another 13 years we will probably be dealing with terabytes.  Why?  Consider this:  The invention of CD-ROMS allowed people to put huge amounts of data onto a CD-ROM.  This sufficed for quite a while, and eventually people saw how much space they had, so the technology advanced to the point of filling the disk.  Then, occasionally, more than one disk was required.

The same thing will happen with D.V.D.  It will suffice for probably a few years.  People will put massive amounts of video into games, instructional applications, or other multimedia endeavors onto one single layer disk.  They’ll see how much space they have left, and continue to push, adding more and more.  Eventually, the dual layers will be full, and things will start coming on 2 D.V.Ds… then there will be something new.

I hope you enjoyed our little trip down memory lane, and our glance into the future.


No more Google Reader?

March 14, 2013

Many times a day, I open up Google Reader. I have used it religiously for several years. it changed my entire web experience. Instead of iterating a list of web sites to see what is new, I changed to use google reader.

They’re retiring it in July. I thought that sudden news may be accompanied by the announcement of a grander new product, but it’s not. It’s simply going away.

It’s a crying shame. I will need to find an alternative product. Any suggestions? Preferably web based…

This will change my web experience again, but this time for the worse.

Excuse me. I need time to mourn.


Attribute Surrogate

February 25, 2013

I’m building an application server that is heavily based on attributes, DI (indirectly via an IOC), and the command pattern.

Basically, you can tell it which “features” to start. It finds those features via attributes (by name), then starts all command processors associated to the feature (also defined via attributes). Each things announces the other things it needs, then those things are found and started.

When you pull an attribute off of a type, you are given an instance of a class. It’s a regular class for which you can use the default constructor, or a constructor that accepts primitive types.

In some cases in the app server, I want the attribute to represent an object that is populated by the DI container. How do you get the IOC (or some other factory) to construct an instance of an attributes?

As far as I know, you can’t. So, I’m using what I am calling an attribute surrogate.

The attribute surrogate is an attribute that refers to a class. The thing consuming the attribute extracts the type information from the attribute, then uses the IOC to construct the other type.

The surrogate attribute:

   1: namespace SomeGuySoftware.ApplicationServer.Contracts

   2: {

   3:     using System;

   4:     using SomeGuySoftware.Common;

   5:  

   6:     [AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = false)]

   7:     public class CommandProviderSurrogateAttribute : Attribute

   8:     {

   9:         public CommandProviderSurrogateAttribute(string name)

  10:         {

  11:             Validators.CheckRequiredArgument("name", name);

  12:             this.Name = name;

  13:         }

  14:  

  15:         public string Name { get; private set; }

  16:     }

  17: }

You see that in this case it has a NAME property that is a string. I started by using a TYPE, which works just as well, but in the case of the app server it’s using the string as everything else within it does. (For more generic purpose, change the NAME attribute to a TYPE).

I slap that attribute onto a class:

   1: namespace SomeGuySoftware.ApplicationServer.Contracts

   2: {

   3:     using System;

   4:  

   5:     /// <summary>

   6:     /// Put this on a feature to register a Wcf service.

   7:     /// </summary>

   8:     [CommandProviderSurrogate("WcfCommandProvider")]

   9:     [AttributeUsage(AttributeTargets.Class, AllowMultiple = true, Inherited = false)]

  10:     [DependsOnFeature("WcfHostFeature")]

  11:     public class HostWcfServiceAttribute : WcfAttribute

  12:     {

  13:         /// <summary>

  14:         /// </summary>

  15:         /// <param name="name">The name of the service to register.</param>

  16:         public HostWcfServiceAttribute(string name)

  17:             : base(name)

  18:         {

  19:         }

  20:     }

  21: }

Now, the app server sees the Command Provider Surrogate attribute and pulls off the name property. It uses the Type Cache (previously blogged about) to find the type by name. (The actual type has a NAME attribute on it which is used to identify and find it). Then, the Type is passed to the IOC which constructs an instance of that type. The IOC does all of the dependency injection, etc, just like it’s supposed to.

Here is the method that does all of that work. IObjectFactory is an abstraction on top of the IOC. As far as the code is concerned, it’s using a generic factory. The implementation of it is an unity container.

   1: private void AddCommandsFromSurrogateAttribute(Attribute attribute)

   2: {

   3:     /*

   4:      The attribute may be a surrogate - this means that the class is decorated with the generic [CommandProviderSurrogate].

   5:      The better practice is to decorate a user-friendly attribute with [CommandProviderSurrogate].

   6:      The user-friendly attribute has properties that can be consumed by the surrogate command provider.

   7:      Example: [HostWcfServiceAttribute] is decorated by [CommandProviderSurrogate].

   8:      HostWcfServiceAttribute is a specific surrogate to start a service.*/

   9:  

  10:     // is the attribute a surrrogate?

  11:     var surrogateAttribute = attribute as CommandProviderSurrogateAttribute;

  12:     if (surrogateAttribute == null)

  13:     {

  14:         // does the attribute have a surrogate attribute?

  15:         surrogateAttribute = Attribute.GetCustomAttribute(attribute.GetType(), typeof(CommandProviderSurrogateAttribute)) as CommandProviderSurrogateAttribute;

  16:         if (surrogateAttribute == null)

  17:         {

  18:             return;

  19:         }

  20:     }

  21:  

  22:     var surrogateType = this.typeCache.GetType<ICommandProviderSurrogate>(surrogateAttribute.Name, false);

  23:     if (surrogateAttribute == null)

  24:     {

  25:         throw ApplicationServerExceptions.CommandProviderSurrogateIsntSurrogateType(attribute, this.feature);

  26:     }

  27:  

  28:     var surrogate = this.objectFactory.GetInstance<ICommandProviderSurrogate>(surrogateType);

  29:     surrogate.SourceFeature = this.feature;

  30:     surrogate.SourceAttribute = attribute;

  31:  

  32:     var commands = this.getter.Invoke(surrogate);

  33:     if (commands == null)

  34:     {

  35:         return;

  36:     }

  37:  

  38:     this.resultCommands.AddRange(commands);

  39: }

For more generic use, have the attribute specify a type, then you can eliminate the TypeCache part of it.


Type Cache

February 8, 2013

At work, over time we have introduced the need to find types in the app domain. This is to eliminate configuration in favor of convention and declaration.. Each of these needs came up independently, and each do it their own ways. That annoys me. it needs to be refactored so that the app domain is crawled only once, and all can share the results.

My latest personal project has the same needs. I created TypeCache to solve the problem.

You tell the cache what types to look for, and also what attributes to look for. Those types are loaded and stored, and the TypeCache is added to the unity container as a singleton. Anything that needs it simply accepts it as a dependency, and away it goes.

   1: public void Blah()

   2: {

   3:     var config =

   4:         new TypeCacheConfiguration()

   5:         .AddType<IServerCommand>(

   6:         .AddType<ICommandProcessor>()

   7:         .AddType<ICommandProvider>()

   8:         .AddType<ICommandProviderSurrogate>()

   9:         .AddType<IFeature>()

  10:         .AddAttributeType<NameAttribute>()

  11:         .AddAttributeType<WcfServiceAttribute>()

  12:         .AddAttributeType<WcfAttribute>();

  13:  

  14:     var cache = new TypeCache().Load(config);

  15:  

  16:     var names = cache.GetTypesThatHaveAttribute<NameAttribute>();

  17:     var wcf = cache.GetTypesThatHaveAttribute<WcfServiceAttribute>();

  18:     var wcfbase = cache.GetTypesThatHaveAttribute<WcfAttribute>();

  19:  

  20:     var services = cache.GetTypes<IFeature>();

  21:     var serviceByName = cache.GetType<IFeature>("name");

  22:     var typesWithWcfAttribute = cache.GetTypesThatHaveAttribute<WcfAttribute>();)

  23: }

I’m still working on the interface. I’m not a huge fan of the load method, but I didn’t want the constructor to initiate the crawl. So, while the details may change, the result is what I need. One thing that will have to change: inject the crawler rather than have it always do the full app domain. (Inspired by MEF catalogs).

The methods return types, usually by name (a custom attribute). Then, I use Unity to construct an instance of the type. It’s pretty sweet.

MEF

I have been somewhat familiar with MEF for a long time. Today I took the time to do my homework on it, wondering if I should have used it instead of inventing the type cache. So far, my answer is no, but I leave the issue open to debate. I want to use Unity for composition, not MEF. If anything, I just want MEF to find the types. I didn’t see anything that would reveal the type. (It can return a subclass of Lazy<T> which may contain the type information… don’t know without trying). For now, I’m going to stick to the TypeCache.


Model On-The-Fly

February 1, 2013

I’m trying to setup a simple way to define simple deployments. This is in support of my most recent project, and also something that can be used by Dev Dash.

Things can be organized like this:

  • A server is one or more server types
  • Each server type may have deployed to it one or more applications
  • Each application has one or more feature
  • Each feature may have one or more web services (and other things, too, but let’s limit it to web services for now)
  • Everything needs files (assemblies, etc)
  • You can group files into lists and add the lists to other artifacts rather than duplicating files

That’s a simple hierarchy. It gets more complicated, though, because you can add things outside of the hierarchy

  • An application has features, but it can also have web services and files assigned to it directly

Dandy. To complicate it more, I want to be able to remove things. For example: Feature A contains 3 web services. But, something above it may want to remove it. IE: Start Feature 3, but exclude one of the web services.

Thus, you end up with a default configuration that you can then add to or remove from.

My first instinct was to start creating some tables to do this, but then I reconsidered and decided to use Model First in EF and let it deal with the tables. Pretty modern of me, right?

image

It’s a whole bunch of entities with many to many relationships. Ugly. Furthermore, it didn’t work. The model shows the associations between the entities. Behind the scenes, those many-to-many relationships become join tables. If I created this myself in the database, then I would add a column to join tables that would allow me to indicate REMOVE. (IE: one entity inherits a property from another entity, but in this instance I don’t want it). You can’t do that in EF other than to build the join tables as entities, which would result in a nasty model.

Another issue is that I want it to be simple The sql to maintain this (lacking a suitable gui) would be ugly too.

Thus, I began a trial-an-error effort that lasted all week. I started to write something to handle this configuration issue specifically, but quickly realized I can do so more generically.

What I came up with is an on-the-fly model. In a single text file, I identify the object and properties I want, populate the objects, and apply rules. This gives me exactly what I need in a generic way, and I will be able to use it for other things.

That said, it remains immature. One thing that I will have to do (should I pursue this) is to define the model first, then populate it after. As-is, it does whatever you want. By defining the model first, rules and relationships can be defined then enforced.

server test1
	servertypes
		add servertype App Server
		remove [servertype App Server]/applications[application testapp]/webservices[webservice service1]
 
servertype App Server
	applications
		add application testapp
		add application testapp2
 
application testapp
	features
		add feature test1
	webservices
		add webservice service1
		add webservice service3
 
application testapp2
	features
		add feature test3
		add feature test4
	webservices
		add webservice service2
 
webservice service1
	files
		add string abc.dll
		add string def.dll
		add string ghi.dll
		add string readme.txt
 
webservice service2
	files
		add string a.txt
		add string b.txt
		add string c.txt
		addlist /[filelist somefiles]/files
 
webservice service3
webservice a
webservice b
feature test1
	webservices
		add webservice a
		add webservice b
feature test3
feature test4
 
filelist somefiles
	files
		add string list1.txt
		add string list2.txt
		add string list3.txt
		add string list4.txt
 
 
 
You can resolve any of the entities that you like, but since I am primarily interested in servers, let’s see what happens when we resolve Server1.
  • It is of type app server, so it gets 2 applications, which results in 3 features and 3 web services
  • It then removes one of the webservices that it doesn’t want.

Everything is driven by ENTITY TYPE and ENTITY NAME.

Entity Type is defined as the first piece of a non-indented line.

Entity Name is defined after the space following the entity type.

The first level of indent is the property names.

The second level of indent is the commands to populate the property.

Currently, everything is treated as an entity except for string. It is coded for specifically. Other primitives will follow as needed.

All properties are assumed to be collections of entities. That too will mature as needed. IE: set the value of individual properties.

Here’s one of the entity paths:

[servertype App Server]/applications[application testapp]/webservices[webservice service1]

That is on the ServerTypes property of one of the Server entities.

  • Finds an entity in the property’s value collection of type servertype and name App Server
  • Goes into the applications property
  • Finds an entity in the property’s value collection of type application and name test app
  • Goes into the webservices property
  • Fine an entity in the property’s value collection of type webservice and name service 1

Finally, it’s where it needs to be, and it removes it.

The ADDLIST command is similar.

addlist /[filelist somefiles]/files

Because it begins with the /, it resolves [file list somefiles] rather than look for it wihtin the property collection. It finds it, drills into the files property, and copies all of the values from the files property of the list to the property that contains the command.

There is one more major thing I want to do: have things automatically collapse into each other. For example: This resolves to server/servertype/applications/features/*. I don’t really care about server type. I want to be able to specify that all of the property of server type are rolled up into server, and we never see server type in the output. But, to do that properly, I must define the model ahead of time rather than on-the-fly. (Otherwise, the collapse rules would have to be defined for each instance of the entity… that’s no good.)

This is how I consume the file:

   1: var stream = new FileStream("ParseText2.txt", FileMode.Open);

   2: var structure = new Parser().GetStructure(stream);

   3: var resolved = structure.Resolve("server", "test1");

   4:  

  • Now, if I iterate the results and dump them to the console:
  • server test1
       servertypes
          servertype App Server – ENTITY
             applications – PROPERTY
                application testapp – ENTITY
                   features – PROPERTY
                      feature test1 – ENTITY
                         webservices – PROPERTY
                            webservice a – ENTITY
                            webservice b – ENTITY
                   webservices – PROPERTY
                      webservice service3 – ENTITY
                 application testapp2 – ENTITY
                    features – PROPERTY
                       feature test3 – ENTITY
                       feature test4 – ENTITY
                    webservices – PROPERTY
                       webservice service2 – ENTITY
                          files – PROPERTY
                             a.txt – 3 strings
                             b.txt
                             c.txt
                             list1.txt – 4 strings added at once from the list
                             list2.txt
                             list3.txt
                             listr.txt

    The next step is to conver that to concrete objects. I believe I will easily achieve that via Dynamic

     


Yes, I’m still here

January 24, 2013

One of the two of you may be be wondering why I haven’t posted any updates about the current project that I’ll probably never finish. I appreciate your concern.

Fret not. This month, despite my better judgment, I have started another major project, motivated by unwarranted visions of grandeur. Things are as they should be: I am spending absurd amounts of time on R&D for what will likely amount to nothing (based on historical statistical data). I just hit a major milestone with it, so I thought it would be a good time to alleviate your concerns. I am still coding.

Regarding my last project, DevDash, it is successful by my meager standards. We have deployed it to several machines at work and use it on a daily basis. I also did a demo of it to a group of people who really didn’t care. That was personally and emotionally rewarding.

There have been a couple deployment related glitches, but they have been mostly due to configuration. Otherwise, it works as it should. It’s pretty cool to drop a DLL into a folder then watch the dashboard as 15 servers download it and restart.

The new project I am working on will eventually absorb DevDash.

As with all projects, even if I don’t finish it (due to boredom), the value is in the education it provides.


Follow

Get every new post delivered to your Inbox.