Showing posts with label development. Show all posts
Showing posts with label development. Show all posts

Tuesday, March 20, 2012

Preparing for Predictive Analytics

It’s been awhile since I have written here because I’ve been heads down on project work and have just had a chance to come up for air. I want to begin with some short posts to discuss a number of challenges that I’m currently working through. The major theme will focus on how companies can prepare for predictive analytics. If you aren’t familiar with the term, here’s a quick link to the Wikipedia topic to get you started.

About a year ago I joined a small company that is focused on delivering top-notch predictive analytics software to address some very specific commercial and educational market needs. Since joining the company as the solution architect, I have found that the single most complex and dynamic part of our engagement and solution delivery process is data acquisition, normalization, and access. This is largely because of the diversity in the platforms, technologies, and applications written for and used by our clients.

We work with our clients to analyze their data for the purposes of building predictive models. The requirements for model building are pretty straightforward. We need a clean and consistent view of the data - requirements that are not unlike any other analytical or reporting process needs. But the complexity of today’s enterprise environments make this more challenging. Additionally, we aren’t looking at a snapshot of data at a single point in time. We are looking at the data in real time or near real time in many cases. We are often working with both structured and unstructured data that come in a myriad of formats and accessed using many protocols. Everything from relational data in databases, to web services, to flat files, spreadsheets, etc. All of the information needs to be identified, cataloged, gathered, date/time stamped, and recorded for time based analysis.

Clients that understand master data management and have sound data governance policies are easier to work with because they understand the value of their data and most importantly how to get it. At the other end of the spectrum are those companies that have their data in many disparate systems, have no data governance or ownership policies, and don’t know the value of their data. Getting access to their data and getting it into a clean and consistent form can be quite a challenge.

Therefore, my next few posts will talk about the challenges we are facing and our approach to solving the data integration and normalization needs for a predictive analytics solution. My hope is that the information you find here will help you prepare for using predictive analytics in your organization to improve and optimize the decision making you do on a daily basis using one of your company’s most valuable assets – your data.

/imapcgeek

Monday, January 30, 2012

You Are Dead to Me

OK, even if Silverlight isn’t completely dead, it certainly is starting to smell pretty bad. Mid last year we were looking at Silverlight as being a viable solution for our UI needs. It has a rich programming model.  The user experience is excellent and it has pretty decent market penetration – certainly not as good as Flash, but respectable.

However, as the majority of our development is greenfield and we are looking to build for the future, it just didn’t make long term sense for us to consider building on a product whose future was looking pretty iffy. When you consider that Microsoft recently cancelled Mix 2012, it’s clear to me that the future lies elsewhere.

Sure, you can still build on Silverlight. And sure it will be supported for quite some time to come. But I asked myself why build on a technology that is clearly questionable in its future? Certainly Microsoft is continuing to support XAML development for Metro style applications. But I questioned whether I was just delaying the inevitable.

In the end, I felt the best decision was to go with HTML5 for the applications we are building for the future. We have begun active development in ASP.NET MVC3 with the Razor view engine. We have achieved our goals of providing a rich user experience, excellent performance, and a testable loosely-coupled code base. It’s also a solution we can work with right now, which means we can deliver business value right away without sacrificing functionality.

v.Next of our applications will give us the opportunity to revisit the scene and reevaluate the options. I think the best news is that there is an excellent set of options out there now and the future looks very bright indeed.

Friday, November 18, 2011

Duplicate MEF Exports When Export Has Metadata

I just discovered that the Managed Extensibility Framework will produce duplicate exports for parts that are exported with custom metadata. My scenario is pretty simple. I have created an Entity Framework DbContext subclass that is marked up with both [Export] and [DbContextMetadata] attributes. DbContextMetadataAttribute is my own custom metadata attribute.

   1: [Export(typeof(IDbContext))]
   2: [DbContextMetadata(ContextType = "Person")]
   3: public class PersonContext : DbContext, IDbContext
   4: {

The result is shown here:


image


I have verified that this is true by simply commenting out the custom metadata attribute. Interestingly, this behavior is not present when using the MEF ExportMetadataAttribute. I’m planning to dig into this a little more to see why it’s happening, but it certainly was unexpected.


 


Blogger Labels: Duplicate,Exports,Export,Metadata,Framework,custom,scenario,DbContext,DbContextMetadata,DbContextMetadataAttribute,IDbContext,ContextType,Person,PersonContext,behavior,ExportMetadataAttribute

Monday, November 14, 2011

How Much UML Modeling Is Right For Your Team Or Project?

This is a question I see asked a number of times in various forums. It is one of those “It Depends” sort of topics, but is one that I think is worth discussing here. In this post, I’ll give you my perspective on how I’ve used it in my projects.

Probably the most important thing you have to understand is, “Why are you using UML at all?” – assuming you are in the first place, or are considering using it in the near term. As the old saying goes, a picture is worth a thousand words. A UML diagram is simply a picture of part of a (potentially) complex system. And sometimes the best way to break down the complexity is through pictures.UML diagrams convey structure or behavior through pictures of classes and their interactions. It supports numerous diagram types to break down the system into various views, which can be combined in order to represent the system from different angles in order to help bring perspective and clarity.

image   image

Organized in a logical way, the UML diagrams you create help to tell the story of the system you are developing. The $64,000 question is, “Who is the story’s primary audience?”. Generally, the answer to that is the development team that is building the application. However, with the success of Agile, many teams feel that UML is unnecessary, or at best, it’s used in ad hoc ways with very lightweight and high level diagrams – maybe even temporarily on whiteboards during meetings. It’s simply viewed as a means to convey a concept or outline. I certainly agree with the spirit of that. UML can help to speed up the team because it helps to bring understanding and consensus through a shared view of what they’re building. How lightweight or detailed your diagrams are, or even if they are persisted, should be a decision you make based on the team’s appetite and ultimately the overall value add to the team and project. Flexibility is the key.

And no matter what the long term goals are, the short term benefit of an increase in velocity is probably the most valuable takeaway you will realize by using UML.

In order to gauge the right amount of diagraming to provide to the team, I have used the sprint retrospectives to reflect on this with the team and arrive at the right answer. The teams that I’ve worked on have all been comprised of a mixture of experience levels. Typically, the less experienced team members or newer team members gain the most benefit from the diagrams. The more experienced team members still benefit from them, though, as they almost always help to disambiguate the design details. The net effect is that it helps to level the playing field across the team and improves communication and ultimately productivity. I have almost always chosen to persist the diagrams in a modeling tool – even if for no one else but myself for future reference.

In a highly Agile team where practices like TDD are used, the use of UML may likely be perceived as an impediment. You can make a pretty well supported argument that this viewpoint is right. The design should emerge from the creation and iterative refactoring of code and unit tests to flesh out the details. Up front design using UML is the opposite of that approach. Unless you consider it from the perspective of general high-level architectural point of view that is. The UML diagrams you use can describe many higher level aspects of the system like: patterns; guidelines on organizing system components; describing the layers of abstraction;  deployment details; or other high-level details. Let TDD do what it’s best at – creating flexible and resilient designs at a low level of detail.

Most development teams have a certain amount of turnover throughout their lifetime. The reasons for this are myriad, but the bottom line is that you will need to bring new team members up to speed at various points in time throughout the life of your project. If the system that they are working on is reasonably complex, the UML diagrams that you created to convey your design concepts to the original team members can be an excellent way to help the new team members understand what they are working on. I am a firm believer that every team member should get the big picture. They shouldn’t be relegated to some dark corner of the application with little or no visibility into how it all fits together. Use of tools like UML helps to bring everyone up to speed, which is a good thing.

A common argument that I have heard against using UML is that like most documentation it will always be stale when compared to the code. I would say that this is generally true. There are many ways to deal with this, but probably the best way is to just accept that fact and understand that the main use of the tool is to help people understand the system. If the diagrams are too far out of alignment with reality, update them. Most tools support reverse engineering. Use it to refresh the details and update the diagrams. Your future team members will thank you. Just don’t get bogged down in the minutia of sync’ing code with models all the time. That’s one of the quickest killers of usefulness.

In the end, I will also yield with an “It Depends” answer for the original question. Since no two teams and no two projects are exactly alike, I’d say you will have to gauge the right answer based on your current circumstances. For me, I’ve always relied on UML as a means to reducing complexity and organize thought around the structure and design of complex systems. Let your team help you to decide the right answer.

Happy Modeling

Smile

Tuesday, November 16, 2010

When Team Velocity is King

Our team met yesterday to do a walk-through on a project that was developed by a group of contractors for a high-priority project. A significant portion of the project architecture relied on patterns like dependency injection, factories, and the like to gain a high degree of loose coupling. This was motivated by the ever changing requirements for the product owner and the demands that the system be easily extensible.

At various points during the review, some of the team members that had the longest time on the team made comments that the design was too abstract, that “we’d never do a system this way.” When we came across a set of tests that had been commented out, the response was, “Good, we didn’t want to maintain tests anyway.”

Being the new guy on the team, I wanted to understand why they felt that way. The general opinion of the team was that abstraction and unit tests were simply too time consuming to implement and yielded too little value to consider for their applications. This position intrigued me – considering how much adoption and support Agile practices for software engineering have across the industry.

I believe this opinion is rooted in the business’s belief that “better is the enemy of ‘good enough’”. They are more interested in getting applications out quickly, with as few bugs as possible, but when bugs do occur, they are VERY tolerant of them. The cost associated with fixing the bugs, even if they are found in the field by end users, is not seen as a significant reason for being more strict in the development methodologies to prevent them.

Instead, they rely heavily on business analysts acting as QA and end users to ferret out the defects that are most critical and fix them then and there. More esoteric bugs that don’t dramatically affect the usability of the applications are glossed over and may be fixed in the future when time allows (or may not).

All of this is motivated by the belief that “going fast is the single most important requirement for our development teams.”

Velocity is King here – and it’s good to be the King.