Wednesday, 10 December 2014

A little bit of powershell in my life

Things were so simple then.

Having used 'nix at University and a bit beyond  I came to love scripting, using things like Bash and tcl,  prior to that I had just written a few simple batch scripts in DOS.    Excluding JavaScript from the fray which doesn't, arguably, fit in the same category as lower level scripting languages I really haven't done much apart from a bit of Perl and  although its called a scripting language it is a high level programming language. 

Years later, and after having been indoctrinated into the Microsoft Stack, I have hardly written any scripts or have had any need to, but that is changing.

ALM


My current place has a few bits of software that use Powershell and, some,  lots of it. Its the tool/language of choice for the ALM framework developed in house, used to push software out from local environments to production  environments (in some cases).  The framework consists of 100s of scripts and XML configuration files, to map to the myriad of upstream test/integration environments and settings required therein,  and  it is plugged into TFS to allow CI and gated/manual deployments (no CD here yet!)

The ALM framework is in the process of being wound down and eventually something like Octopus deploy is going to be used in its place, with the ability to provide extensibility - ironically - using Powershell.

There are lots of arguments against the bespoke framework  including things like why a company whose competitive advantage is not ALM software didn't use an off-the-shelf product, for one.
Apparently there weren't many (ALM frameworks/tools) about when a consolidated and consistent release framework to production was originally sought.    There are aspects of it I like, but as it stands it is far too fragile and "indie" to be a serious candidate for continued use, but that's mostly a moot point.

Regardless, it has given me the chance to use Powershell and can be used really productively for the jobs you hate.

Turning on our distributed computer


As part of a re-platforming exercise, we are currently undertaking, of a large e-commerce application, often referred to as the old legacy ball of mud.  We are ripping  it to shreds (into a new world glass of sand?) and  refactoring large parts of it into a distributed messaging based system. 

NServiceBus is the technology of choice to provide pub/sub choreography and all of the benefits that come with it (resiliency, durability, scaling, error retries and  SLAs to name but a few) ,  With this however, comes a new set of problems over an in-process monolith, not least administering the many smaller parts deployed out to n-number of bits of tin. Which is where this rambling is going toward making my point (sorry!)

Due to licensing issues, and with a consideration for allowing easy scale out, there is a need to host and administer lots of windows services (hosting NServiceBus endpoints)  on lots of different servers.  The ALM framework does this most of the time, but not always,  leaving little choice of going through the rigmarole of waiting another 15 minutes to push out deployment, or turning on the bits needed out on the boxes.  

NServiceBus provides "endpoints" which are effectively message handlers listening out for specific messages from other endpoints they have subscribed to or are being routed messages from (i.e. events and commands). Once they receive a message they will do something as a result, like persist some data or start off a process.  

Considering that a typical arrangement of NServiceBus endpoints involves something called a distributor (dishing out work)  on 1 box and 2+ workers (actually performing the work) on other boxes, its easy to see that if there are 40odd services in total, then in total  there are potentially 120 services (housing the endpoints) which are going to have to be administered. As you can imagine, administering start/stop gets very tedious, very quickly in the services manager and this is where Powershell comes into its own.

For example: 


The simple script above starts all the services on 3  the boxes named in the $computers array, which have the "OrganisationRootNamespace" in the name.   This saves about 15 minutes of clicking and lots of sanity.

The stock library of Powershell has lots of commands providing easy programmatic access to common OS tasks such as registry editing and copying files.   Before settling in for a big point and click-a-thon  consider using scripting, for the jobs you hate.  

Another thing to consider Powershell for is if you are automating deployment of software components, there are often  requisite tasks that are needed,  adding users to user accounts,  copying files and perhaps pulling code from source control for example.

Powershell comes in handy again here.  If you want to do more with Powershell you can load in managed .NET DLLs and have access to powerful APIs for things like TFS and Active Directory.   

It can be argued that this can all be done in any .NET language, but 
scripting is appealing for getting stuff up quickly, easier adoption/learning curve for noobs, and no need to use a *clunky* IDE. 







Tuesday, 18 November 2014

Messaging and transient issues.

Messaging and transient issues.


A small post but something worth considering. 

Recently when discussing a situation where a MongoDB replica set was in the process of failing over, concern was raised about writing data - whilst a new primary was being elected. 

This is going to be a transient issue and issues similar to it - such as temporary server outages and routing  are too .  

They are going to take a little time to resolve but should resolve fairly quickly. In the meantime, there are a few solutions available whilst this transient issue sorts itself out:  

  1. Do nothing.  In this case give up and find another job  you lazy hacker.
  2. Let the process fall over, report  the failure to users and let them try again via a button click. A users experience might be sullied - in the opinion of some -  but it still could be a reasonable way to recover (this depends on what stakeholders/business think really).  This might not be reasonable for important information which must stored and can not be optionally retried. 
  3. Employ a retry mechanism, simply loop a few (reasonable times) until we get success, or employ an exponential back off to give reasonable time for recovery.  I have done the former using, as someone I know put it, some funky "AOP" shit.  However, I wouldn't recommend doing the AOP stuff for transaction management or bounded retries, because eventual consistency still may never occur and AOP, certainly in a lot of the frameworks I have used, is complete magic. I did have trouble explaining the concepts to some of the less experienced and even seasoned members in my team what that "Transaction" attribute was above my DAL method, and how this was setup on the IOC container and  how code was executed pre and post the method in an execution pipeline .
  4. Use a durable messaging framework like NServiceBus.  The command or event (message) which was sent will fail and if second level retries are enabled will be retried a specified number of times. If unsuccessful within the specified number of retries (which have exponential back off) The message will be placed on an error queue,  relevant administrators will be notified or at least be reporting on this stuff.  The exception and or problem will be noted and hopefully fixed and then the message in the error queue will be replayed bringing everything back into a consistent state.  And all of this with the user completely unaware that four data centers where nuked.

That is all. 

Async/Await

Async/Await

As I am sure you are aware, this is not a new feature in .NET - having been around for over a year as part of .NET 4.5. There have been plenty of posts about it, and so I'm not going to go into a great deal of depth, as I am not an expert and there are people who have gone into it in more depth than I could.  I just want to get a couple of  the key concepts up and provide a laymans tilt on it. 

I've been using the language feature as part of a site I am developing as a platform to testbed a few technologies including:

  • Knockout.Js, 
  • The continuous integration and continuous deployment capabilities of  Visual Studio Online (formerly Team Foundation Service)
  • WebAPI and integrating with 3rd party web services
  • Some of the newer C# language features 
  • Bootstrap templates 
  • Stanford Core NLP for .NET 
  •  
The site is work in progress and is available @
http://wearedev.azurewebsites.net/


The main purpose of the site is as a front end for a simple aggregation service for searching for development related articles, plumbing in information from a number of sources such as YouTube, MSDN, PluralSight and Stackoverflow.   I've got a few ideas about potentially leveraging the Stanford Core NLP  fpr .NET ported by Sergey Tihon so that I can, perhaps, perform slightly more accurate content matching, but its a bit of a mess at the moment. 

What is Async/Await? 

 

A C# language feature making it almost trivial to perform asynchronous work.  There are a number of different ways to already do this in C#/.NET including the Asynchrounous Programming Model (APM), think IAsyncResult, and Event  based Asynchrounous Pattern (EAP) think Background Worker.  I'm not going to discuss these much here, but suffice it to say async/await is probably a better and easier way to do async work and is also now the preferred way to do async work (I believe there are some low level optimisations to take advantage of here provided for by the TPL (Task based parallel library) which async/await sits atop of.    

In order to identify a method as something which will initiate asynchronous work,  we let the compiler know by adding the async keyword to the signature like so:



Here the Evaluate method is going to initiate some asynchronous process. It also makes this method awaitable. That is to say, a call to this method  will not block and can work in an asynchronous manner too. 

The Evaluate method will return a Task<ResultSet>.  The return type here is important.  We are not returning something that will necessarily have the value immediately but rather something that signifies that the result of some work will appear at some point in the future.    I think I've heard this same kind of concept being called a promise in other languages.  Another thing to note is that the actual return statement inside the method body will wrap the return type in Task<T>. So here ResultSet is wrapped in Task<ResultSet> impicitly.

Inside the method a call to _dataFeedRetriever.GetDataFeedResults(searchTerm) is prefixed with await.   This means that when the aforementioned method does work which does not get a result immediately,  the Evaluate method, itself, will be returned from immediately,  but crucially the position, in the Evaluate method, where execution got to, will be retained/saved so that when there are results available, normal synchronous execution will resume inside the Evaluate method.

  
Interestingly prefixing a method call with await, has the effect of unwrapping the Task<T>  type the method returns and pulling out the results when they are available.  Conversely, prefixing the return type in the signature means that the return type of the method is wrapped in a Task<T> implicitly. 

If  awaitable methods are nested,  when encountering the first await where a delay is encountered, execution returns immediately on the first await and then where each await is encountered, back up the call stack, so we back out upstream if you like.  When a result is available at the inner most await, execution resumes here and then will continue back up the call stack, perhaps without any further delay at the outer awaits. 

A misconception of async is that utilises additional threads to do its bidding, but this is not always the case.  A single thread can be used to  call out to an async operation,  such as file IO (i.e. I/O bound work).  Whilst IO work is being done and a response/result is being waited upon, the thread, used to make the request, can be freed and returned to the thread pool to be used to perform other work, such as incoming web requests.   Once the IO is completed, the thread, upon which the operation was started, can be scheduled to be used to continue execution where it left off.  

This kind of pattern allows for more efficient use of system resources and threads and potentially reduces the need to create new threads  helping save on all of the associated costs with this (TEB, TKO, User mode stack  and context switching).

Facilitating the above is a state machine which the compiler generates where the async/await keywords are encountered.   It is framework magic but I think it safe to say this is no fly by night 3rd party library but a C# language feature, which although not infallible, is pretty consistent.   

Coincidentally, the state machine which is generated for async/await is not too dissimilar to the state machine used for the yield keyword, where the position in a sequence is maintained whilst enumerating a sequence.   Jeffrey Richter leveraged the yield feature to provide an async library before it became part of the C# language.   If you are interested about Jon Skeet has both an excellent tutorial where he dissects async/await EduaAsync and also a Pluralsight video (which is probably a little easier to follow)  going into the innards of the state machine used in async/await.

Putting it all together

 

Using async/await in the Website mentioned earlier allows me  to query  the 4 feeds I mentioned at the same time,  and take advantage of only having to wait as long as the longest call to a feed, as opposed to the sum of the calls to all the feeds. At the same time making use of only one thread to do all this work and in a nice clean easy to understand manner, is a great argument for using the feature.   I'll add the rest of the code to this post or the next. 

Sunday, 16 November 2014

MongoDB

Mongolicious


MongoDB training, provided by MongoDB themselves, was up for grabs recently, so I put my name in the hat to get a keener insight into MongoDB and NoSQL databases, having used them only a few occasions. The training was attended by a near 50/50 split of devs and DBAs and this led to some interesting debates and reactions.  

Like lots of the stuff I've been introduced to/have started looking at recently (I am late to nearly every technology party there is) they (Mongo and NoSQL) have been around for a while and have become established as a viable alternative persistence solution to the likes of MSSQL. 

The main features:
  • Its schema less
  • Document oriented/No support for JOINs 
  • Querying performed using JS/JSON
  • Indexing
  • Replication/Redundancy/High availability
  • Scaling out via Sharding 
  • Aggregation (unfortunately we ran out of time so this is not covered here)
  • Authorisation (again ran out of time on this one) 

Its schema less


You'll know doubt be aware that relational databases use schema to determine the shape and type of data that is going to be stored in them.  MongoDB doesnt, quite simply.  You take an object from some application (most likely) and it is serialised (typically) from your application into  JSON and then into BSON and then stored on disk.  

MongoDB does not care for the type of data used or its shape and when a document is added, a field with a particular datatype,  used for one commit, may be of a different type on another (I'm not sure how this would impact indexing)  But the look of sheer horror and gasps from the DBAs was priceless (more on this in a minute). 
In practice data type changes like this would probably be as rare as they are in an application as they are in the database, how often (apart from during development) do you go and change the fundamental data-type (that is,to say not its precision or  max size) in production code/databases? 

Document oriented


As there is no schema,  things we think of as objects in applications (e.g. a Customer object)  can be serialised into JSON and then as a MongoDB document added to a collection of customers.  A collection is roughly analogous to a table and therefore a collection can have a number of documents like a table can have a number of rows. 

Documents can be entire object graphs with other embedded documents (e.g. a customer with a collection of addresses).  Access to documents can be expressed through dot notation when querying and projecting.  

With this arrangement the notion of relationships expressed  through queries over many sets of tables (like in SQL) dissapears, relationships are instead expressed by virtue of parent child relationships you see in a typical object graph.  Joins are killed off at the DB layer and would generally then be performed in the application (if at all).  However,  there is the ability to define foreign key relationships with other documents in the database.  There is a cost to this though as referential integrity will come at some cost to query performance.  

With document orientation though I think it seems like a good opportunity to embrace and leverage small transactional boundaries perhaps closely aligned to an aggregate root/entities like in DDD.  Each document commit is atomic and so getting the aggregate root root right, I'm sure, would certainly improve concurrency and reduce  the size of working sets.  

Querying performed using JS/JSON


One of the particularly great features of MongoDB is the simplicity of its querying. 
For example finding all of the documents with a customer whose name is bob on a reports database (db context is reports below) within the customers collection looks like:



The find method is simple JavaScript the first argument is JSON - straightforward. The predicate is expressed by a key and value pair, with the key being the field in a document with values matching the specified value.  Just  like in SQL, complex relational expressions can be built up using in built functions such as $lt, $gt, $or and $in.


Taking it a little further we can determine which part of the  document we would like to view


The second argument to the find method {customer_name : 1} is a filter to say which fields in the document to return in any results found, analogous to the select statement in SQL.   Here customer_name will be returned. We could inverse the statement passing 0 to customer_name and all of the fields EXCEPT customer_name would be returned. 

As queries get more complicated there can be a tendency to unbalance the brackets in the expression, so be mindful of this.  A good tip was to add in all of the sub-expressions first and expand them out.  Alternately, use a text editor which has highlighting for brackets/curly braces etc.

Indexing

Indexing is supported via single or compound indexes on document fields.  They can be configured differently on primary and secondary nodes dependent on needs, with a bit of jiggery-pokery (taking nodes offline and configuring them differently).  
I can't speak much for the absolute speed and performance of the indexes, but relative to not having them, they are much quicker. 
Under the hood they are implemented using B-Trees, whether this is optimal would be for some performance analysis and comparisons to answer. 

Replication/Redundancy/High availability


Apart from when development sandboxing and running in standalone mode,  the recommendation is to run MongoDB databases in a replica set which must contain at least 3 nodes to be effective and always an odd number for larger replica sets (explained in a moment). A replica set consists of a minimum of 3 nodes in total.   A primary and 2 secondaries.  

All writes and reads are, by default performed, at the primary and then the instructions and data are replicated to the secondaries.  

Primary node failure and subsequent failover is handled by an election process to determine which secondary should become the new primary.  The odd number of nodes in a replica set is crucial here as this ensures that a majority can be obtained by participating nodes and a new primary elected.  If an even number of nodes was configured in a replica set, there could be a  shared vote for leader and all of the nodes would go into a read only state, as there would be no primary.  I'm not entirely sure if MongoDB enforces the odd number rule we didn't try it out in the examples.

In the 3 node replica set arrangement, if  the primary node where to fail, an election would take place between the secondaries and which ever node obtained the majority vote first would be come primary. Obviously, nodes can not vote for themselves.  In this instance there is 50/50 chance of either one of the secondaries becoming the new primary, as long as there are no priorities configured. 

Using secondaries here as foils for the primary would see this arrangement more attuned with Mirroring in MSSQL where exact copies of the primary data a farmed out to slaves (in essence) ensuring redundancy.   And so replication here is not necessarily the  same as replication used in MSSQL.  

However, a slight tweak to configuration of the secondaries means that all reading need not occur through the primary, and specific secondaries can be read from directly.   This provides an element of scaling out, where writes and reads are performed through different servers,   Indeed, indexing can be tuned separately on any of the nodes to help with optimal query execution

The configuration of replica sets is an absolute doddle and this was the general consensus amongst both devs and DBAs.


Scaling out via Sharding


This got the trainer hot under the collar and is one of the more prominent features, particularly for the larger software projects you may find in an enterprise. Sharding only requires a  tad more work  than is required for adding a replica set to get going.    

Analogous to partitioning in MSSQL, sharding is, in essence, a way of making use of a distributed index.   It also helps distribute storage and retrieval across MongoDB instances, thereby performing a kind of load balancing. 

Sharding occurs at a collection level and an appropriate shard key is used to shard upon. So, for example, if we were to take a customer collection we could shard it on the email address of customers.

Consequently documents with email addresses starting A through S and S through Z could end up being placed into 2 different shards with all read and writes for data with email addresses falling within the respective ranges being routed appropriately to write or retrieve related data (a mongos server and configuration servers are required for this) 
  
Shards contain chunks of a pre-defined size.  If data, in a shard, exceeds this threshold then splitting occurs and data is moved between shards in attempt to maintain a close to equal split of data amongst the shards.   This process occurs in the background, it will most likely impact performance but its not something the user has to worry about (that is to say this would be a non-functional requirement).  This allows auto-scaling to a degree, until you realise or plan for more shards as your data needs grow. 


DBA Horror

I think the developers found using and configuring MongoDB a more edifying experience than the DBAs.  I would hope the DBAs involved would agree I was in fair in saying they seemed to find the use of the shell, command prompt and JSON a little challenging.   Higher level concepts similar in MongoDB and SQL worlds were easily digested by attendees from both sides.  The consensus though, generally, was that MongoDB was a low friction route into administering and working with a highly scalable and available data persistence mechanism for developers!   Having only been a SQL hacker and not being involved enterprise persistence choices I'm not best placed to say whether MongoDB would be a  better or replacement solution for SQL, but it does seem bloody easy to use and administer.

To this end I would recommend attending the MongoDB course if it wings its way into your organisation or take the MongoDB intro courses on Pluralsight or MongoDB University to help make an informed decision before defaulting to SQL.



 


Monday, 1 September 2014

TFS and integration test builds from Shelvesets.

Our team have been involved in a fair bit of release activity, as of late. We are gearing up for our wagile (our previous CTO's term not mine) project's big bang release to production.

Due to the release pipeline we have been forced to release our changes with another team.   

The code resides in 2 different branches, both of which are branched off of a main branch.  However, in TFS terms, these are unrelated until at least one merge has taken place (a baseless merge). 

Quite simply we have the following arrangement:

a (main)
b (us)   c (them)

It was decided that we should merge our changes in (b) into their changes in (c) and then use the build resulting from the merge to push out to an integration environment, where our testers would run any remaining manual tests (or kick off any semi automated service/integration tests) 

Forgetting the fact this would actually be a baseless merge in the first instance, actually performing the merge and getting it out,  would mean that we would either have to push out locally from where the merge was performed and not check-in, or  - if we were prepared to rollback if anything was to fail -  check in.  
We took the latter option in the and it was a bit painful, as there were failures and we had to rollback.

To save this pain we took advantage of the fact that builds can be pushed out from shelvesets as well as normal changesets and latest sources.

Bearing in mind a merge still has to be performed , the merge can be checked-in, as a shelveset, and then this shelveset can be referred to in the options on the build definition like so:



Simply choose "latest sources with shelveset", in the drop down,  and then the relevant shelveset and voila you have branch (c) +  branch (b), without any of the baggage of a changeset  (I'm sure internally shelvesets are  stored in a similar manner but are surfaced differently.)   


Hopefully this, "soft merging" saves someone a bit of pain and allows integration testing before fully merging 2 branches.

DDD or not to DDD? That may or may not be the question.

DDD

I've had/am involved with my first commercial experience of DDD, or at the very least, something heading that way. My team are developing their first DDD software component.  I didn't realise I was at least 10 years behind the leading edge ( the year  which Eric Evans seminal work DDD was published was the same year I started my career as an embedded software developer, which is as close to enterprise development as Pluto is to the Sun)   

DDD Quickly

After being in a few larger organizations and having dealt with a fair few types of design approaches (ranging from monolithic to typical n-tier), it seems  that DDD has finally achieved mainstream adoption in the industry and the shit kickers - I include myself here -  are just about ready to consume it, and murder it potentially :) 

I have read "DDD quickly" on InfoQ, which I would highly recommend to get a flavour for DDD in general, and have recently started reading  "Implementing DDD" by Vaughn Vernon (or Vince Vaughan - as I keep saying) .  I chose this over Eric Evans' book as it is more current and seems to have lots of practical examples. The book is well written and has so far (3 chapters in) kept me  engaged and interested. Pretty good going for quite a dry subject like DDD. 
 
My first impressions of the subject are that its complex,  nuanced and subjective.  When things aren't black and white and there are lots of possible ways to skin a cat,   extra attention and leaps of faith are required.  Also because of this there are lots of terms to learn and use which in turn refer to almost every other aspect of important literature in the field of enterprise 
development (P of EAA, GoF, DDD) .  I will never be able to grok it, I have too many distractions, like life.  However, I will try and learn it enough to practice it and contribute to discussion about it. 

DDD Lite

For our team's problem at hand it appears, on the surface,  we have had some of the hard work done for us and have a  bounded context with a ubiquitous language that we can play around with. Additionally  the bounded context is conveniently mapped 1-1 to a sub-domain (good practice according to Vaughn).

The lack of any analysis about what the bounded context and sub-domain are here, by our team, and also our use of just a few tactical patterns (aggregate/entity identification and the use of a repository) owe to the "DDD lite" subheading, which Vaughn says is symptomatic of a lack of strategic patterns being employed.  

That is to say our team is taking a technical approach to DDD and is not fully involved in bounded context or ubiquitous language identification at this point. (or other strategic practices). This is fortunate and unfortunate.  Fortunate that we can kick on and use tactical patterns but unfortunate in that we can't apply the strategic patterns which we will have to apply at a future point in time and thereby miss out on crucial learning in this area.

The software component was initially going to be developed  using an anaemic domain model/data model and transaction script. However, our architect saw it as a candidate for DDD, with the idea that it will prepare us for future DDD projects.
 

I don't believe there was anything particularly wrong with the familiar and fairly straight-forward approach (anaemic/transaction script), we were going to use, and going on some of Vince's guidelines (or Greg young agile-esque T-shirt size - Small Small estimation), the component wouldn't necessarily be a priority for DDD love, as it is not in a core domain or a particularly complex sub-domain, but I have to agree with my architect, it does seem that it is useful as a stepping stone as our team gear up for more DDD development.  

The bounded context is a little lacking though, there are no invariants,  and consistency is a fairly straightforward affair.   There is no usage of domain events, either, for example.  However, the structure of the component means it aligns to a ports and adapters type architecture, where we can plug in back ends and front ends in with ease  (for example we plug in a resilient/distributed messaging framework  to communicate with other bounded contexts).

Back to OO

Martin Fowler describes the anaemic model in less than favourable terms.  I have seen it used in my enterprise and elsewhere.  I now know where I have seen it correctly used, in simple RESTful CRUD components for example, and "wrongly" used if what you end up doing is  pulling behavior into a myriad of "domain?" services  -  which is arguably an easier way to get to work done on some data - and transposing/pivoting behavior out of  domain objects and splaying/fanning  it into "services", relying on DI to pull everything together. If your constructor has 15 dependencies maybe something is a bit smelly.

So with DDD what happens regarding anaemia?. We feed the domain model up by falling back on encapsulation, a fundamental tenet of OO, where state and behaviour - on that state - are packaged together, heping form what is known as a rich domain model. 

We are told it is a big problem that this kind of stuff has been abandoned for the skinny latte way of design, but I do wonder how bad it actually is.   


The Dilemma

So now we are developing using DDD (or "DDD lite") and this goes back to the question in this post's title.  Should we or shouldn't we? It seems that this decision shouldn't just be based on technical considerations or that the decision should just be outright rejection because of hearsay about complexity, over engineering or the absence of a highly-skilled hero team.  Vaughn states that DDD is a complex process but out of it can drop a clear and simple manifestation of a solution to a set of  business problems. 

There are a series of questions that should be asked before embarking down the DDD route which people like Greg Young and Vaughn Vernon both advocate asking.  One of those shared by them goes something like: "By embarking on DDD for your software, is this adding value to the competitive advantage of the business the software is being developed for?".  

In this instance we are adding value to the competitive advantage of the business - even if it is marginal.  We are also providing ourselves with a firmer footing for DDD work in the future, which will see further value added.  So maybe that's OK even if we are just doing "DDD Lite" for now.  

If adopting only some of the tactical patterns give us marginal improvements this is  still worthwhile and not hurting the business, and if an opportunity like this on a project you are working on presents itself - to get a leg up  with DDD without going the whole hog - then it should be considered.










Friday, 4 July 2014

Currying and Closures

Just a quick one

Closures might be well known amongst many C# practitioners - apart from  me it seems. I read about them ages ago and then haven't knowingly applied them.  Of course, I have been using them  (or at least a variation on the theme) often in C# by virtue of using 'lambdas in lambdas' when using LINQ for example, and lambdas which refer to variables in the method in which they are created. 

I mentioned functional programming including Currying in the last post, and this is still my current focus, its satisfying my ADHD for the time being.  

Anyway, It turns out that if you want a bit of a better understanding of how Currying works,  then understanding closures can help you  take a leap of faith. 


Currying in Haskell

The following example is taken from Learn you a Haskell

The function multiplies 3 numbers together and returns the result.

The  first line defines a function, multThree,  which can be read as  taking 3 arguments and returning a value - all of the same type - signified by a. 

The second line maps the first 3 arguments to  x,y and z which are then used in a function body where x * y * z occurs.   When the function is called, e.g. multThree 2 5 5, it will yield the result of 50. There is no need for an explicit return , like in C#, the result is just returned.

In Haskell, every function (well, nearly - I think - functions can actually be uncurried) is curried automatically which means that functions with multiple arguments are broken down so that each argument has an accompanying return value.   This opens up the door to Partial Application  (to be covered in a future post), which means that not all of the arguments need to be supplied immediately.  

Taking the above function definition again but showing where Currying occurs with the use of parentheses:  

We have a function which takes an a and returns
a function which takes an a and returns a function which 
takes an a and returns an a.  Confusing,  yes.  Bloody confusing. 

This is how I have read Currying definitions in lots of places which hasn't been a great deal of help.  What I couldn't work out, particularly, was where the actual multiplication would take place, if state was being maintained somewhere and/or if the result was being worked out cumulatively (via some kind callback mechanism maybe).

Reading

After reading a few chapters of a CLR via C#, a mixture of blogs and stack overflow posts,   a simple rearrangement of the function makes things alot more clear. (as well as knowing closures are at work)

When the multThree function definition is encountered by the Haskell compiler something like this occurs:

Three lambda statements are created.  
The first takes x and 
    returns a lambda which takes y and 
       returns a lambda which takes z  and has the function body which returns the result

(Note the  "/x -> ..."  notation, this signifies a lambda that takes one argument in Haskell, in C# this is very similar  " x => ...." would mean the same thing.)

And with values: 

With all of the values known, the function body can multiply all of the values together, to get a result.  And this is the important bit, only when ALL of the values are supplied can the function body be executed.  

The key thing here is that this is all facilitated by closures which was the missing bridge to working out how the values were getting passed around.  Basically each nested lambda or child lambda closes over its calling environment and any of the values present in its calling environment are available to it.  Essentially the values, once supplied,  are in scope all the way back up the calling chain.   

Back to C#

In trying to work all of this out I cross referenced an example by Jon Skeet on Currying and Partial application and a chapter in  CLR via  C# by Jeffrey Richter about lambdas,  which showed how lambdas referencing variables in the method, in which they are defined, close over that method. That is, have access to values in the method. 

There is a curious side effect in C# (at least)  in that a value which is closed over, has its scope widened beyond the method to the class in which the method is defined. This is because  a helper class is generated for the lambda method - containing  the implementation of the method itself and values which that method will need to use which are copied into the class.  This class is then added to the original class containing the anonymous method as a field.

I then cross referenced that this occurred in Haskell (well on a few S/O posts - so still anecdotal - but certainly reasonably explained)  and this has cemented my understanding.  

Off to the Balti House

Currying may be straightforward but without knowing some of the facilitators it can be hard to know  or accept, even, what is supposed to happen when Currying is occurring.  A lot of the posts I have read regurgitate, almost, the Wiki definition of Currying which isn't that helpful if you want to know a bit more of the "how".   I might have simply needed to read somewhere about lambda calculus  which explains this simple requisite.  But in case you land here, before there, then I hope it might help a little. 

Monday, 23 June 2014

Functional programming and blind adoption of 3rd party software.

Training

I was lucky enough to have had the pleasure of attending an introduction to Domain Driven Design, CQRS (coined - in part at least - by the presenter himself)  and Event Sourcing by Greg Young at Skills Matters in London last week.   It was snappy, contentious - at times - and very informative and definitely gave me  a lot of food for thought, I now have a reading list as long as my arm.    As an added bonus, the people who attended, as well as Greg himself, were approachable and easy going, as the beers after will testify to. 

The background to me getting on this course,  is that we are currently going through a massive set of architectural, infrastructural and cultural  change at my current employer, and this course has set me up to be asking more searching questions and provide input on some of the stuff we are currently doing and are planning to be doing soon.

I'm not going to write about any of the above though, a quick search will elicit a million articles, and will be a lot more informative about them than I could be, at the moment. 

Piqued Interest


However, the thing that interested me most was a small nod to functional programming.  I'm aware of functional programming but didn't study it at Uni and didn't really know the differences between imperative and functional programming until I actively searched it out.

Greg mentioned that he was using Erlang which has been  developed by a phone company for future use internally and this had given him ideas about how to approach programming in the imperative world, from different angles.  I was hooked, anything that can make me think about problems differently is worth knowing about.  


Containerless

Everyone knows, by now, that the best (and maybe only) way to arrange your n-tier architecture is to introduce your favourite IOC container after separating out your dependencies and introducing myriad mapping between the layers.  You are now at the pinnacle of software architecture.  Or not, perhaps.  

The problem with introducing a container, or any 3rd party library for that matter, is reliance on that bit of software being stable or reasonably straightforward to leverage.  When they are neither or worse, life becomes painful.   Greg raised this point and it really resonated, as I have been there before. No matter though, he suggested a way around it!


Container Bowels

I  have been in the bowels of Unity (of Enterprise Library fame)  trying to work out why I'm  getting a Stack Overflow exception after cycles in my composition root (where the dependencies are registered) have been inadvertently configured (obviously only knowing that bit retrospectively).  Whilst at the same time the job I'm actually supposed to be doing is on the back burner. 

The solution.  Ditch the container.    I had searched for alternatives registering dependencies without a container, in the past, but gave up after about 10 minutes of Googling and didn't think to attempt it any other way.  I know it's not a great leap to do all the wiring up in one place but its easy to become blinkered when something, in the main, seems to work even if it is a painful from time to time.


Partial Application

Partial application is a technique used in functional programming which allows a function to have some of its arguments to be made invariant or hard-coded in effect.  This function can then be used as an argument to another function giving the same result each time it is invoked.  This allows composition of dependency trees . For example, a function which took a connection string, which returned a data access component could be used by multiple functions which required data access.  (Code example to follow) 

This approach can be used as a way of  performing dependency injection and allow you to get rid of the container and see all of your dependencies up front in one place without 10k lines of code acting as an intermediary.   You will inevitably lose some flexibility and features but on the other hand you may gain simplicity, performance increases and less headaches, so make a judgement call; you might just have defaulted on this before. It will certainly make me think twice.


Haskell

After learning about this I wanted to know a bit more about functional programming and came across Haskell and an absolutely fantastic tutorial website Learnyouahaskellforgreatgood.  The first few chapters are fairly easy going and  I am now up to the section about Currying (related to Partial Application)  and it was only at this point that I had to refer to my mother tongue of my day to day programming (C#) in order to understand exactly what was going on, as I couldn't quite visualise or translate it.  

Step forward Jon Skeet and his distillation of the concepts in C# Jon Skeet currying in C#.  This helped to highlight a potential issue or advantage, perhaps, I  end up trying to transpose one language into another. If I was starting programming from a clean sheet I wouldn't suffer from this turmoil. 

Haskell is based on lambda calculus and its easy to see it was used as an inspiration for C#'s LINQ  - which has seen "lambdas" being awarded first class citizenry by the language designers

And so it dawns on me that by using  LINQ  and lambdas I have been using (at least in essence) functional programming without even knowing it.     

 I particularly like the heavy use of recursion (not always optimal but definitely elegant) the quick sort example (although noted "cheesy" by the Haskell tutorial website author)  shows just how elegant Haskell is and, shown side by side with a C example, is about 10 times smaller!   

Collections and their processing are a fundamental tenet of Haskell and simply splitting lists into heads and tails makes you reason about where that approach might have been useful before.  I am yet to get to folding but I'm hoping this and other features,  gives me some new ideas about   how I can approach problems in the imperative world in the future..   I've still got lots to get through and it may just turn out to be an academic exercise but its better than Coronation Street. 

So...

I intended this post to be far more technical, with code examples, but I can't find my notes from the course and so the presentation of some examples is a bit lacking.  The main thrust of the post  though, I suppose, is to consider the adoption of 3rd party libraries a little more carefully before folding them into your architectural cake mix, I've been burnt on more than 1 occasion and so I won't just blindly follow implied industry practices or being a technology magpie hankering after the latest and greatest.














Friday, 23 May 2014

SSDT March 2014 Static Code Analysis Extensibility

Static Code Paralysis.

Coding standards (and more latterly styles) in any organisation of any size are liable to incite religious wars and makes zealots out of even the most considered or apathetic of us.   Inconsistency and subjectivity,  lack of enforcement and heavy handedness all contribute to the contempt often leveled at them.  

Sod it.

I say sod all that, if you or your company aren't using static analysis tools  then you are missing out on improved code quality and productivity (borne out from a consistent approach) and reducing the most banal and mind numbing conversations and sometimes pissing contests associated with discussing the minutiae of standards.  

As an aside, having a few years under my belt  in the industry  I've found that people who "really" moan most about this are usually rabble-rousing morale sappers who like to moan about everything, so it stops them doing this and maybe might lead to them getting on with work instead!

Makes things easier for everyone involved - really.


Barring any major clangers or anything truly detrimental to program performance and readability if there is a consistent view of coding standards and styles which are automatically enforced,  and can - within reason - be suppressed and challenged from time to time to aid with best practice, then the pain of the standards largely goes away and it becomes something that just happens is accepted and then more important things which should be reasoned about -  are.

Currently, in our organisation we use a combination of  StyleCop, Resharper, and FxCop for our .NET static code analysis needs and its generally a pretty edifying experience, even though there were a few integration issues which were quickly resolved. 

The SQL parallel universe.


There is one big place (at least) where all of the pain and fundamentalism, mentioned above, still exists, the database Universe.   Our team have taken the initiative to get code analysis into the enterprise for new SQL development in order that our data team has less reasons to pull releases, saving many thousands of pounds and delay in the process, for missing a few spaces and tabs (yes this stuff really does still happen).  

Moving the identification of this type of "defect"  back to the development life cycle, where it belongs,  in itself saves cost, egg on peoples faces and confidence from the exec that the IT department aren't a gang of idiots. 

SSDT Code Analysis Extensibility

With the SSDT March 2014 release, comes the ability to extend (once again - its been missing from the last few releases) the code analysis rules for Visual Studio 2012 and 2013, that you get out of the box.   

The MSDN example application for extending the rules is good  (Authoring an SSDT Rule the accompanying documentation for the some of the types used in the API could be better, which meant having to debug to find out what some of the TSqlFragments were, which was a little time consuming.
  
A very important thing to note is that you will need to install the SSDT 2014 March release in order to debug and execute the code analysis sets you develop.  Also the folder which is probed for the extensions is in the SQL extensions folder 
  • <Microsoft SQL Server Install Dir>\120\DAC\Bin\Extensions
NOT the Visual Studio extensions folder 
  • <Visual Studio Install Dir>\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\Extensions
I couldnt get the latter location to work after a number of attempts. 

That being said, once the sample is up and running getting yourself going is a breeze.  

The analysis is facilitated by dog-fooding  SQL scripts under scrutiny, into a set of strongly type TSqlFragments  which are themselves plopped into an AST.  The TSqlFragments  allow for extensibility via use of the visitor pattern, accepting TSqlFragmentVisitors which works really well.  

This arrangement also allows for fine grained control and the ability to have scope, such that a BooleanExpression on a JOIN can be treated differently to one in a WHERE clause, by nesting visitors.  

The visitors are used by rules derived from SqlCodeAnalysisRule which can look at the fragments and allow developers to codify their organisation's take on what they should look like or whether or not they should be present.  For example, ensuring SELECT * isn't used or that column numbers aren't used in ORDER BY clauses.  

Errors and warnings are presented to users by building up a SqlRuleProblem giving appropriate context and severity. 

The TSqlFragmentVisitor has over 800 overloads which, incidentally,  rips ReSharper intellisense to shreds. It allows developers to visit on nearly every conceivable element type of a SQL script, and do something with it. 

Developing rules can be time consuming especially when considering specific formatting rules.

Its quite conceivable that a tool could be written atop of this API to code gen the rules, especially the formatting ones. This would be a great toy project.  

As I write this I have just come across TSQL Smells by Dave Ballatyne
TSQL code smells in SSDT 
There are 43 rules (compared to the stock 15 or so in Visual Studio) you can utilise, it is open sourced and can be extended or used as a leg up for your own stuff.

One last advantage and compelling reason to use SSDT is the fact it will work with msbuild, with little or no extra config other than ticking a tick box and ensuring the library is accessible on your build agents,  a large consideration when assessing your needs. This will ensure CI is subject to the same static code analysis as local development envs.

In Summary.

SSDT isn't the last word on SQL static code analysis, there are other tools available such as a Red Gate offering, SQL Enlight and SQL Code Guard to name but a few (and there really are only a few) but all of these have been judged (in relation to our needs),  after recent research by my colleagues to have various shortcomings (from price, incomplete features, to lack of integration in Visual Studio) in comparison to SSDT. 

Given all of this it seems that SQL static code analysis is still an immature area and hopefully one which will see more attention in the not too distant future, and hopefully I don't have to author too many rules....