RPC is Working

December 5, 2014

Just a quick update: The service api, client and server, now supports request/reply messages.


It’s just quick POC code, nothing shippable. It doesn’t have a timeout, which it needs, incase there isn’t a service running on the other end.

There is one other messaging solution that I want to implement that isn’t built into RabbitMQ. That will probably have to wait a while though. I need to start replacing a lot of this quick hacky code with real code.


RabbitMQ Service Layer – Publish

December 2, 2014

Let me preface by saying the RabbitMQ code is pretty bad. It’s just enough to get it working. The RabbitMQ implementation of the service layer isn’t much more than POC.

Anyway: the idea is to host and consume services, all ignorantly. As a developer writing code to be hosted in the app server, you shouldn’t need to know how the services are being hosted, or how to call them. Today the app server does it using RabbitMQ. Maybe tomorrow it’s using Service Bus for Windows or Web Api. I want all of that abstracted away leaving the developer with simple interfaces.

Here’s a test feature called DemoFeature3. The service “TestService” is associated to the feature. The service starts and stopped as the feature starts and stops.


Here is that service:


It’s not the most interesting method I’ve ever written. It has 2 simple parameters, and it simply displays them. Whoopty-doo. Notice, though, that there’s nothing special about this class. It has the optional NAME attribute (which is redundant in this case), but that’s it. It’s just a class. It doesn’t even have a service attribute. We know it’s a service because the feature said it is. (Although that may change going forward. Currently I only have one messaging pattern implemented. Attributes may be needed to specify things as more patterns are added.)

Also notice that there is only one method. That’s just laziness. You can put in as many as you want. At this time, though, you cannot have 2 methods of the same name. The binder is simplistic and would have to be improved to support overloads.

When DemoFeature3 starts, the service starts, and it will start listening for messages. The RabbitMQ implementation (which is completely hidden from this code) starts listening on the queue for this service. The queue name and exchange name are known by convention, and will be created as necessary.

What about the client? It’s RabbitMQ, so it’s just a message. You could submit a message from the RabbitMQ admin page if you’d like. Or, you can do it in code. If that code is in the app server (or using it’s client library), then it doesn’t know about RabbitMQ either. The message you send must be a JSON serialized ServiceRequest object.

Loose Typed Client

So far, I only have a loose typed client. I would like to create a strong typed one, but that’s going to require some thought and, ultimately, on-the-fly code generation. That’ll be a big chunk of work, and it’s not on my real short list, but I will get to it. (or, maybe you would like to get to it for me? Still looking for help people!)

The service name, operation name, service type, and parameters are all wrapped up into an object called ServiceRequest. The server picks that apart and uses the container and reflection to instantiate the class and invoke the method.

There is an API to do this. You can assemble a ServiceRequest yourself. It’s just a POCO with some fluent methods to make it… fluent. But, generally, that’s not necessary and you can use a simple helper method.

What follows is AnotherPeskyFeature. This was used previously to demonstrate the local scheduling feature. You put an attribute on a method that says “call me every second”, and the app server will call it every second.

I enhanced it to make as service call every time the method fires.


See that the service client is injected in the constructor. Now, you can use the service client to invoke the service: service name, operation name, parameter 1, parameter 2. This is the disadvantage of it being loose: Like non-hateoas REST you need to know what to call.

The IServiceClient interface does not have the method shown above. Instead, that is an extension method of IServiceClient that takes the parameters, converts them to a ServiceRequest, then calls the method that is defined by the interface. I did that because the implementation of that method will never change regardless of the backend technology, so there’s no reason to have multiple implementations doing exactly the same thing.

In this demo, both features are in the same application, which is hosted by a console app. Thus, you see each feature creating it’s output. Even though they are running next to each other, they don’t know that they are.. One feature is putting messages on the queue, and the other is taking them off.


If you deployed the application to multiple machines, then all of those machines would consume the same queue so that you can spread the workload.

Enabling It

There’s nothing to do on the hub. On the node:


I can easily add an extension method to do all 3 at once. The BUS and SERVICE LAYER have nothing to with each other, though. You can eliminated the UseRabbitMQBus call, and the buss will use it’s SignalR default. (I don’t have a second implementation of the service layer.)

Rabbit MQ Details

I’ve written my own quick API on top of the RabbitMQ API. “Quick” is the key word. I just need it to work. I can spend weeks working on that alone.

The exchange names and queue names are determined by convention. Currently, the client sets up the exchange and queue if needed, but I would prefer the hub do that on the node’s behalf to reduce the rights required by the nodes.

There are 4 sets of credentials: default, bus, service host, service client. If you don’t set specific credentials for any of them, then the default will be used.

Here’s what this stuff looks like in the portal:


The application feature “DemoFeature3” depends on core feature “ServiceHost”. Thus, it won’t let you shut down ServiceHost unless you shutdown  DemoFeature3 first.


If both are stopped, then ServiceHost will start automatically when you start DemoFeature3. Sweet.


To host a service: add a ServiceHost attribute to the feature, and pass it the type of the service. You don’t concern yourself with RabbitMQ or any other technology that might be in play. As an application/feature developer, it’s not your concern.

To send to a service: Add an IServiceClient parameter to your constructor, then use it to invoke the service.

What’s Next?

Request/Reply. While it should be avoided when possible, you know that you’re going to need it. The user will invoke the method. The service client will send the request and wait for a response. (In RabbitMQ, this is done via a one-time use response queue.)

Also, like I did for Web API, I will provide the option to have a service also be a feature. That way, if all you want is the service, you can do it via single class. That’s minor.

Swapping out SignalR for RabbitMQ

November 26, 2014

Out of the box, the app server uses signalr for 2 things:

  • The bus – it is how all nodes communicate with each other
  • The website – it is how the website communicates with the bus

Since the beginning, I have always said that “anything in the app server can be swapped out… theoretically”. I always added the “theoretically” disclaimer because I hadn’t actually tried it.

Finally, I tried it with RabbitMQ, and the theory was incorrect. I had to make some changes, and chose to make some others.

The primarily required change: the two things done by SignalR were handled by a single class (with a TODO to break it up.) If you switch the bus to RabbitMQ, then you still need SignalR for the website. Managing those 2 things as one doesn’t work. So, per the TODO, I broke it up.

Secondary required change: the battle for flexibility and simplicity continued. There are server level features and application level features, and they all need to be easily managed via code. I want everything to work out of the box, and then also be modifiable either via configuration on the hub, or via code overrides. The real trick from the code perspective is that the applications are, essentially, remote. Any changes you make need to be made to the server then propagated to the app. I spent a lot of time working on that, and it’s now in a good state. (Ironically, for the purposes of RabbitMQ, I don’t even need that yet. But, it’s done. You can manage application features and server features from the server api.)

Chosen change: On the hub, the SignalRBusHub being hosted as a feature in an application.  It was quick and dirty. The feature simply subscribed to the bus then forwarded messages to SignalR. On the application, though, the SignalR connectivity was built into the bus object itself as a subclass. So, that was inconsistent. I changed the hub level bus to also be a subclass hosted by the server. That cleaned it up.

Chosen change: The SignalR code was all within it’s feature. It wasn’t reusable. I changed that so that SignalR hub hosting is a core feature that other features can opt into. That was already implemented for WebAPI and LocalScheduler, but the ability did not exist when I wrote the SignalR code. The implementation of it was very similar to WebAPI. That’s not because they are both ASP.NET technologies (that’s just coincidence), but because they both involve features registering with other features. The WebApi Feature keeps track of how many other features require it, and manages itself based on that. Anything that uses the OPT-IN approach will need this. I took most of the webapi feature code and moved it up to base classes, and implementation of the signalr feature became pretty insignificant. To register another feature with the signalr feature is just an attribute.


The SignaRHubHost attribute says that “when this feature is started, start the BusHub SignalR hub.”

Finally, I had to write the RabbitMQ code. There’s a lot of work to be done on that, but it’s working. (IE: a bus message can expire after a few seconds. Currently, it doesn’t.) At the moment, it’s the bare minimum.

Now, you can simply say UseRabbitMqBus(), and signalr is swapped out for RabbitMQ. (You are reminded that this is a test console app, so don’t pay attention to the static fields, etc.)


And here is the implementation of it.


The funny thing about that is that it’s specifically disabling SIGNALR. What if you also have a third implementation? To solve that I will introduce a FeatureRole, and then allow for disabling of the entire role. Then you’d be able to do: server.DisableFeaturesOfRole(“BusHost”), or similar.

For SignalR, I wrote the code to receive messages and delegate to clients. For RabbitMQ, I don’t need to do that because RabbitMQ does all of that automatically. So, there isn’t a server component running in the app server for RabbitMQ. It’s just the client bus.

The following screenshot is the admin portal as one of the demo features starts. The feature fires a bunch of useless child activities just to show how features can send messages that are associated to parent activities. In this case, during the start activity it creates a child activity that fires a bunch of messages for no good reason. I was pleased to see that, as expected, the message numbers are all in order. You do not get guaranteed message order, or guaranteed delivery, with SignalR. They are mostly in order, but there are always a few that are out of sorts. With RabbitMQ, they are in order every time.


I’ve put a lot of time into this over the last few days. I’m really happy to have RabbitMQ working via a single method call.

This is a good update. Now that RabbitMQ is in the mix, I can start working on the service layer. The service layer will need an out-of-the-box default for when you’re not using RabbitMQ. Usually I do out-of-the box first, but for this I will use RabbitMQ first because I’m looking forward to diving into it. But first I need some sleep.

Would you like to write some code?

November 20, 2014

I’d really love some help if anyone has some free coding time. I have specific things that need to be done. Some big, some small. Some concrete, some vague and requiring research.

If you’d like to participate in this project that will almost certainly amount to nothing (love the enthusiasm), then please ping me.

One of the smaller/concrete things is to rewrite my TypeCache object with a better fluent interface. We can then wrap that up as NUGET package. The TypeCache needs a better name. What it really does is serve as a library for convention based programming. Currently, it’s mostly attribute based, but we can expand it to account for type names too.

The app server uses it very frequently to find mostly classes and some methods based on their metadata and types.

Creating and Starting an Application Server

November 20, 2014

The app server gets everything it needs from an IOC container. The IOC container is configured via extension methods, and may also be tweaked manually if necessary.

I have a console app that I use to host a hub and two nodes. This is my primary tool for testing. The console application itself isn’t important, so disregard the use of static fields, etc.


  • new UnityContainer() – I choose unity. Theoretically, you can use any container you want by writing an IDependencyResolver adapter for it. But, that is untested.’’
  • ToDependencyResolver() – that’s the unity adapter of IDependencyResolver
  • AsHub/AsNode – sets up the container with all of the default stuff it needs for each role. For a node, you pass it the server id object. For hub, it will create one on it’s own as the name is known to be HUB.
  • CreateServer – instantiates a server. This is very simple. It’s just new Server(serverId)
  • Start – an awaitable method that starts the server
  • Result – the server

AsHub/AsNode can be combined with CreateServer to do both steps at once. But, then you’d lose tweaking capabilities. Once you initialize it AsNode, you can then change whatever you want in the container before creating the server.

I will likely add a method such as CreateNodeServer() that combines both steps if you don’t require further configuration.

There are two other methods:

  • AsApplication – this is used for the applications hosted within the application server. This is done by the BRIDGE that links the app server to the applications.
  • AsStandAlone – with this configuration, it doesn’t use a hub. It gets it’s configuration, etc, locally. This worked when I wrote it, but i haven’t used it in several months.

Remote Installer, Round 2

November 14, 2014

A key feature of the app server is the ability to install it on a remote machine via the admin website. You fill in a form, hit submit, and the server will be created on a remote machine.

Information required:

  • Installer credentials: the user that will run the install
  • Service credentials: the user the service will run as
  • Remote machine name: where to install to
  • Drive letter: which drive to install it to. (So far, you can’t select the folder, just the drive. The folder is determined by convention.)
  • Serve Type: optional. Select how the server should configure itself.
  • The name of the application server. (There can be multiple per machine, but the name must be unique across the network.)

The hub runs in IIS, which is why we need 2 sets of credentials. IIS isn’t an admin user and doesn’t (or shouldn’t) have enough access to the other machines to install software.

The steps of the install are:

  1. Deploy the files to the remote machine
  2. Grant LogonAsService right to the service user on the remote machine
  3. Create the service on the remote machine
  4. Start the service on the remote machine

When I created the installer, I was still in feature demo mode, and just wanted to get it to work. I thought the quickest way to do that would be to generate a batch file and execute it. Process.Start allows you to pass credentials, so I thought i would use that to execute the batch file as the installer user.

It was a long time ago. I don’t remember the details of all that went wrong, but it turned into a disaster. I ended up with 2 batch files: one to run on the hub, and another to run on the remote machine. I had to use PSEXEC to run those batch files, one locally, one remotely.

After much shenanigans, it worked, but it wasn’t the quick fix I was looking for, and also it was really ugly. But, it worked, which is all that I was after at the time.

The reason I went through that is because I didn’t want to deal with writing all of the impersonation code that would allow me to execute C# code as a different user.

Finally, late last week, I circled back on that and settled in to deal with the impersonation. I properly nourished myself and made mental preparations. Then, all of a sudden, I realized it’s 2014, and we have at our disposal a little thing called NUGET. I quickly found and added the SIMPLE IMPERSONATION library, and there we go… I had impersonation.

And it works great. I haven’t had to play with anything. I’m fully confident that I could write it myself, but I would’ve tripped along the way. It’s nice to have it just done and work. So thanks for that.

Now, the installer is much simpler. I deleted a lot of code.

  • The file deploy is done via C# code. There is a configurable list of files. It iterates the list and copies the files.
  • Grant LogonAsService is accomplished by executing NTRIGHTS.EXE from code. NTRIGHTS.exe can run against a remote machine.
  • CREATE and START the service are accomplished by executing SC from code. SC also works against remote machines.

I’m using PROCESS.START three times. As it evolves, it would be nice to eliminate that and do it all in code, but it’s MUCH better now. I’m able to collect the error output and standard output and save it to the response object.

I haven’t done this yet, but now that it’s back in code, I can use the bus to publish messages as the install proceeds. Those messages will be received by the installer page, which will display the messages to let you know what’s happening as it happens. (Today, it just sits there until the install is complete. It’s boring.)

Finally, this change offered another huge benefit: It’s so much faster. PSEXEC took a long time to run, and I ran it multiple times. It would take 20 or 30 seconds to install. Now, it takes less than 300 milliseconds. That sounds ridiculous, so I confirmed it a few times.

So far, it’s only working as a unit test.  You can kick it off from the page, but you won’t get any feedback just yet.

I have to make one more change. Previously, you would install the app server, then set the server type as a second step. I recently changed that so you that you can, optionally, specify the server type on the setup page. But, I cheated and had the controller set the server type before calling the installer. It’s a hack. The installer needs to be self contained and do it all itself.


Here I use my super awesome REALLY pattern. The code that calls this method does error handling and impersonation. Impersonation is only done if demanded by the installer request. That method is pretty ugly, so I’m not including it in order to protect my self esteem.


This method is called three times: NTRIGHTS, SC CREATE, SC START.


The installer response was previously specific to the batch file implementation, and was marked as a todo. This one is more generic. You add a series of messages to the response, and the message indicates “OK”, or “PROBLEM”, and any user details.

This information will ultimately be published and, possibly, recorded.

Currently, I’m handing uninstalls manually as needed. But, that will have to be coded, too, at some point.

Application Server: Hosting Services

November 7, 2014

I am at QCON SF this week, an like last year, there is a whole lot of talk about microservices.

The application server supports Http services via WEB API. It’s a minimal implementation at the moment. If you associate your feature to an ApiController, then when the feature is started, the ApiController becomes available.

    public class DemoFeature1 : Feature
			public Task OnStart() { .... }
       public Task OnStop() { .... }

The, YabbaDabbaDooController is just an ApIController.

public class YabbaDabbaDooController : ApiController
        public DateTime GetNow()
            return DateTime.Now;


Now, the ApiController is bound to the feature. If the feature is started, you can reach the controller. If the feature is stopped, it will respond with a 404 (if any other controllers are started), or it won’t generate a response at all (if the webapi feature isn’t running because nothing needs it.)

The thought process behind that is that there may be several things related to a feature. Scheduled jobs, web api endpoints, custom code, etc. The feature is the in-app container that manages that stuff as a unit.

But, what if you only want to do one thing, and that one thing is host an API Controller? Then you have to create the feature, create the controller, then tie them together. That’s now awful, but you’ll have 2 classes for every service. And, when thinking about microservices and the possibility that there’s going to be many of them, it can be done easier.

My brain has been churning non-stop thinking about this. I had no intention on working on it, but this morning it got the better of me and I accidentally woke up 2.5 hours earlier than I normally would. So, I figured I might as well get to work on it.

I had to make fundamental change to features, which worked out better all around. To implement a feature, you subclass the Feature class, then provide Start and Stop methods, even if you don’t need them. The feature manager uses the TypeCache and DependencyResolver to find and build features based on the understanding that they are all of type Feature. Then, there are attributes on the feature to indicate what type of feature it may be. Currently, Core, Server, Application.

What I want to be able to do is this:

    public class HttpServiceDemo1 : ApiController
        public DateTime GetNow()
            return DateTime.Now;

It’s a single class with a single attribute. It is the feature and the functionality associated to the feature. Of course, it can’t inherit from ApiController and Feature, and I didn’t want to do a bridge, so the base class had to go. I could have switched it to an interface, but then if I would feel obligated to put the Start and Stop methods on it, as I am not a fan of marker interfaces. And, if Start and Stop are on the interface, then you’d have to implement them even if you don’t need them. Now that I have been doing this a while, I find that often they are not needed, such as in HttpServiceDemo1 class.

So, it’s based on the attribute. The feature manager looks for all types that have an attribute that inherits from FeatureAttribute. Depending on where the feature manager is (primary app domain, or child application), it looks for either ServerFeatureAttribute or ApplicationFeatureAttribute (from which Core and Http descend).

If you need a start method, then add a “void Start()” method. If you need a stop method, then add a “void Stop()”  method. The methods will be found regardless of their access modifier.

Support of all this required some changes. The typecache/dependency resolver stuff had to be changed to work off the attribute rather than the type or type name.

  • The TypeCache had to be updated to return descendants of the requested attribute. Previously, when I said “give me all types that have the ServerFeatureType attribute”, it would only return those that had exactly that attribute, not any of it’s subclasses.
  • I added a method to the type cache to retrieve a type of a particular name that had the requested attribute.
  • The code that calls the OnStart and OnStop methods has been changed to use reflection. The method names have been renamed to just Start and Stop. Previously, these were awaitable methods. That has been lost during this change, but can be put back if needed. But, the feature manager was just waiting on them anway, so no loss. (Because it’s now using reflection, multiple Start/Stop signatures can be supported.)

The first success was eliminating the feature class and get everything working as it did before. But, it wasn’t done quite yet.

The Feature manager would recognize the type as a feature because it has a Feature attribute. But, how does it figure out that that the feature needs to be associated to another feature without another attribute? In this case, we want the HttpService1Demo class, which is an API Controller and also a Feature, to be associated to the WebApiHostFeature, which is what actually hosts the controller. Previously, that was done by adding a [WebApiHost] attribute to the feature, which we can still do, but the point of this exercise was to eliminate as much as possible.

Thus, the HttpFeatureAttribute class was made.

    public class HttpFeatureAttribute : ApplicationFeatureAttribute
        public HttpFeatureAttribute(StartMode startMode = StartMode.Started)
            : base(startMode)

        public override IEnumerable<Attribute> GetOtherAttributes(object attributeOwner)
            if (!attributeOwner.GetType().IsSubclassOf(typeof(ApiController)))
                // TODO:
                Debug.Fail("need to be an api controller");
                return null;
            return new List<Attribute> { new WebApiHostAttribute(attributeOwner.GetType()) };

The GeOtherAttributes method is new. It allows the attribute to accept an instance of a class, and return additional attributes for that class. In this case, it’s returning the WebApiHostAttribute.

The RegisterFeatureWithOtherFeatures method of the feature manager is responsible for the child to parent relationships. I added the second block of code to pull in the additional attributes that are provided by the Feature attribute.

            // this is a super hack. Consolidate with the method version.
            // get the attribute assigned directly to the class.
            var classAttributes =
                    .GetCustomAttributes(state.Feature.GetType(), typeof(DependentFeatureAttribute), true)

            // the feature type attribute of the feature may indicate that additional attributes are needed.
            // IE: the HttpFeatureAttribute returns a WebApiHostAttribute so that features of that type are automatically associated
            // to the web api feature, without requiring a hand-added attribute.
            var featureTypeAttribute = (FeatureTypeAttribute)Attribute.GetCustomAttribute(state.Feature.GetType(), typeof(FeatureTypeAttribute));
            var other = featureTypeAttribute.GetOtherAttributes(state.Feature);
            if (other != null)
                classAttributes = classAttributes.Union(other.OfType<DependentFeatureAttribute>());

And that’s it! Now, you can simply put an HttpFeature attribute on an ApiController, and it is a self-contained feature and feature implementation. Of course, you could still associate the feature to other sorts of things if you want, but the point of this was to quickly stand up a service, as a feature, when you don’t really need anything else.


Because HttpServiceDemo1 relies on WebApiHost, you cannot stop WebApiHost unless HttpService1 (and any other WebApi dependent feature) are stopped first. You can hit the stop button, but it will respond and tell you that you can’t because of the dependencies.

If WebApiHost is stopped, and you start a feature that requires it, then it will start automatically.

One thing that i didn’t do, which I only just now realized looking at the screen shot, is give the new feature attribute it’s own group id. The UI orders thing by Group id, which is why all of the core features are always at the bottom of the list and in blue. I can set the group id for the new HttpFeature attribute and have them all grouped together too, and add it to the CSS so that they all show up as muave or something else distasteful. It’s all easy stuff.

In conclusion, this all went quick and easy, and Features are now simpler and cleaner. The thing that started it all was the idea of being able to stand up a service real quick, and that has been achieved. You may use it for a microservice, or any other type of service you like.

If you like the old way of doing things, then you can do that too. The old way is better when your feature does many things. The new way is better if you just want to host a service.