Thoughts on Open Source, Data Services, and Integration

SnapLogic 2.1 is now available as a Beta release, and we encourage everyone to download and test it out.

The most visible change in this release is a complete reorganization of the SnapLogic Designer, to improve navigation and editing of transformation resources and pipelines, The new implementation uses a tree control that improved navigation for larger repositories, and an improved search capability. The available editing area is is now much larger, and the pipeline canvas now supports ’snap to grid’ capability.

Behind the scenes, there are quite a few improvements as well. We have enhanced the change detection in the server and designer, to simplify development and maintenance. When changes are made, the Designer now highlights the affected transformation resources in any affected pipelines, and allows viewing or automatic application of the changes to the pipeline.

The change detection is intelligent, and will only highlight relevant differences in view, parameter, and pass through settings. A popup provides detailed information about the changes, and allows the changes to be automatically applied to the affected pipeline.

Other improvements include help links from the general properties page directly to the documentation, support for duplicating transformation resources across servers, better management of multiple servers and runtime statistics directly in the Designer.

As always, the installer images for Linux and Windows make installation painless, and take care of all the dependencies. The SalesForce.com, NetSuite, and SugarCRM extensions have all been updated for 2.1 compatibility, and the Windows version now runs as a full windows service.

Loraine Lawson has an interesting post over at IT Business Edge that highlights a recent Forrester Research study on the Future of Software as a Service Technologies. In the study, Forrester predicts minimal success for BI and Integration delivered as SaaS.

This news is no surprise to me.

Back in late 2007, Goldman Sachs released "Getting SaaS savvy - successful investing in on-demand", a research report on SaaS from an investment perspective. (There’s a summary of the Goldman report at SandHill.com). Goldman used 10 criteria to assess applications and determine whether there are well suited for a SaaS model (and potential investment) . Among those, the amount of integration needed was a key criteria. BI and data warehousing fared poorly. Integration on demand was not included in the report, but using the same criteria, it would not have been done well either.

This news isn’t "bad news for SnapLogic" either. As a software vendor, we have always believed the customer should be able to control their deployment, so SnapLogic runs portably on multiple platforms, virtual machines, and Amazon Web Services.

There’s definitely ambiguity around the term ‘SaaS integration’. Integrating on-demand applications is not the same as on-demand integration; there are big differences between the two.

The decision on where to run the integration process should involve the location of the data, security constraints, and the customers preference.

If the bulk of the data is already on premises, then integrate by pulling the necessary SaaS data on-premises. Security wise, this approach doesn’t introduce major new issues related to the data.

If the bulk of the data is SaaS, then a hosted or on-demand approach to integration may be more appropriate. Security wise, it may not be possible to send on-premises data out, and that has to be taken into consideration as well.

One size does not fit all in data integration; there are too many changes afoot in the IT landscape, and everyone need to have options.

Gartner has released their 2009 Cool Vendors in Data Management and Integration report, and here at SnapLogic we’re thrilled to be named as one of the innovative vendors.

In the report, Gartner notes the expansion of data integration requirements to "encompass data beyond the boundaries of individual enterprises". One of their recommendations is to:

Investigate data integration technologies that can unify data residing in applications both inside and outside an enterprise’s firewall, in order to address alternative models for adopting business applications (such as SaaS) with flexible, yet cost-effective, approaches to data management.

Data integration has always been a complex problem. There are some good reasons why, including identifying and mapping data, defining the business rules behind transformations, managing data and metadata, and the fundamental nature of a process that crosses organizational and functional boundaries.

Changing requirements have always contributed to integration complexity. However, the current wave of industry changes is more fundamental; not only are we crossing organizational and functional boundaries, we now are crossing the enterprise boundary as well. Once you cross the enterprise boundary, you’re in the realm of cloud computing, with all its opportunities and challenges.

The cloud computing movement has put the core IT landscape itself in flux. These changes to the IT landscape are having a direct impact on data integration is three ways :

The JumpBox team have released a new JumpBox for SnapLogic.

JumpBox simplifies server software deployment by providing pre-configured packages for software applications, which can be deployed on multiple virtualization and cloud platforms. Since we began SnapLogic, we have always believed that users should be able control their own deployments of software, and JumpBox is another great example of that flexibility.

JumpBox adds support for Parallels, Microsoft Virtual PC, Virtual Iron, and others to the existing SnapLogic platform matrix, and also simplifies configuring SnapLogic on our existing platforms, like Amazon Web Services.

You can also run an online trial directly from the JumpBox site without downloading or installing anything.

The JumpBox team also created a great example of using SnapLogic to check contacts in SugarCRM against the Specially Designated Nationals (SDN) list published by the Office of Foreign Assets Control.