Showing posts with label modeling. Show all posts
Showing posts with label modeling. Show all posts

Tuesday, March 20, 2012

Preparing for Predictive Analytics

It’s been awhile since I have written here because I’ve been heads down on project work and have just had a chance to come up for air. I want to begin with some short posts to discuss a number of challenges that I’m currently working through. The major theme will focus on how companies can prepare for predictive analytics. If you aren’t familiar with the term, here’s a quick link to the Wikipedia topic to get you started.

About a year ago I joined a small company that is focused on delivering top-notch predictive analytics software to address some very specific commercial and educational market needs. Since joining the company as the solution architect, I have found that the single most complex and dynamic part of our engagement and solution delivery process is data acquisition, normalization, and access. This is largely because of the diversity in the platforms, technologies, and applications written for and used by our clients.

We work with our clients to analyze their data for the purposes of building predictive models. The requirements for model building are pretty straightforward. We need a clean and consistent view of the data - requirements that are not unlike any other analytical or reporting process needs. But the complexity of today’s enterprise environments make this more challenging. Additionally, we aren’t looking at a snapshot of data at a single point in time. We are looking at the data in real time or near real time in many cases. We are often working with both structured and unstructured data that come in a myriad of formats and accessed using many protocols. Everything from relational data in databases, to web services, to flat files, spreadsheets, etc. All of the information needs to be identified, cataloged, gathered, date/time stamped, and recorded for time based analysis.

Clients that understand master data management and have sound data governance policies are easier to work with because they understand the value of their data and most importantly how to get it. At the other end of the spectrum are those companies that have their data in many disparate systems, have no data governance or ownership policies, and don’t know the value of their data. Getting access to their data and getting it into a clean and consistent form can be quite a challenge.

Therefore, my next few posts will talk about the challenges we are facing and our approach to solving the data integration and normalization needs for a predictive analytics solution. My hope is that the information you find here will help you prepare for using predictive analytics in your organization to improve and optimize the decision making you do on a daily basis using one of your company’s most valuable assets – your data.

/imapcgeek

Monday, November 14, 2011

How Much UML Modeling Is Right For Your Team Or Project?

This is a question I see asked a number of times in various forums. It is one of those “It Depends” sort of topics, but is one that I think is worth discussing here. In this post, I’ll give you my perspective on how I’ve used it in my projects.

Probably the most important thing you have to understand is, “Why are you using UML at all?” – assuming you are in the first place, or are considering using it in the near term. As the old saying goes, a picture is worth a thousand words. A UML diagram is simply a picture of part of a (potentially) complex system. And sometimes the best way to break down the complexity is through pictures.UML diagrams convey structure or behavior through pictures of classes and their interactions. It supports numerous diagram types to break down the system into various views, which can be combined in order to represent the system from different angles in order to help bring perspective and clarity.

image   image

Organized in a logical way, the UML diagrams you create help to tell the story of the system you are developing. The $64,000 question is, “Who is the story’s primary audience?”. Generally, the answer to that is the development team that is building the application. However, with the success of Agile, many teams feel that UML is unnecessary, or at best, it’s used in ad hoc ways with very lightweight and high level diagrams – maybe even temporarily on whiteboards during meetings. It’s simply viewed as a means to convey a concept or outline. I certainly agree with the spirit of that. UML can help to speed up the team because it helps to bring understanding and consensus through a shared view of what they’re building. How lightweight or detailed your diagrams are, or even if they are persisted, should be a decision you make based on the team’s appetite and ultimately the overall value add to the team and project. Flexibility is the key.

And no matter what the long term goals are, the short term benefit of an increase in velocity is probably the most valuable takeaway you will realize by using UML.

In order to gauge the right amount of diagraming to provide to the team, I have used the sprint retrospectives to reflect on this with the team and arrive at the right answer. The teams that I’ve worked on have all been comprised of a mixture of experience levels. Typically, the less experienced team members or newer team members gain the most benefit from the diagrams. The more experienced team members still benefit from them, though, as they almost always help to disambiguate the design details. The net effect is that it helps to level the playing field across the team and improves communication and ultimately productivity. I have almost always chosen to persist the diagrams in a modeling tool – even if for no one else but myself for future reference.

In a highly Agile team where practices like TDD are used, the use of UML may likely be perceived as an impediment. You can make a pretty well supported argument that this viewpoint is right. The design should emerge from the creation and iterative refactoring of code and unit tests to flesh out the details. Up front design using UML is the opposite of that approach. Unless you consider it from the perspective of general high-level architectural point of view that is. The UML diagrams you use can describe many higher level aspects of the system like: patterns; guidelines on organizing system components; describing the layers of abstraction;  deployment details; or other high-level details. Let TDD do what it’s best at – creating flexible and resilient designs at a low level of detail.

Most development teams have a certain amount of turnover throughout their lifetime. The reasons for this are myriad, but the bottom line is that you will need to bring new team members up to speed at various points in time throughout the life of your project. If the system that they are working on is reasonably complex, the UML diagrams that you created to convey your design concepts to the original team members can be an excellent way to help the new team members understand what they are working on. I am a firm believer that every team member should get the big picture. They shouldn’t be relegated to some dark corner of the application with little or no visibility into how it all fits together. Use of tools like UML helps to bring everyone up to speed, which is a good thing.

A common argument that I have heard against using UML is that like most documentation it will always be stale when compared to the code. I would say that this is generally true. There are many ways to deal with this, but probably the best way is to just accept that fact and understand that the main use of the tool is to help people understand the system. If the diagrams are too far out of alignment with reality, update them. Most tools support reverse engineering. Use it to refresh the details and update the diagrams. Your future team members will thank you. Just don’t get bogged down in the minutia of sync’ing code with models all the time. That’s one of the quickest killers of usefulness.

In the end, I will also yield with an “It Depends” answer for the original question. Since no two teams and no two projects are exactly alike, I’d say you will have to gauge the right answer based on your current circumstances. For me, I’ve always relied on UML as a means to reducing complexity and organize thought around the structure and design of complex systems. Let your team help you to decide the right answer.

Happy Modeling

Smile