SiCortex

Something strange is happening at SiCortex.  A friend says the bank has taken control and laid off the staff.  I hope they aren't forced into bankruptcy -- it would be very bad for HPC.  Their machine is unique in its low-power approach to supercomputing... kind of like an IBM Blue-Gene but with lots of memory per core, and a 500Gbit/second network on top of that.

Keep your ears open, probably an announcement today or tomorrow.

And I was on the last step of installing CPSeis on their Houston machine.  It was SO EASY!  Fribbits.

Microsoft's Cluster Server 2008

I listened in on a webinar last week hosted by RAI and Microsoft.  Being a Linux fan for many years, I was skeptical of what the new Microsoft Cluster server could do -- especially since my team had evaluated it a couple of years ago and thought "ho-hum".  I will say they really surprised me with the speed at which they have taken this "also-ran" operating system and made it into a viable challenge for the Linux geek community to seriously consider.

If I were managing a new HPC operation I would give it serious consideration.  They had an issue with the networking interconnect speeds -- it's fixed (30% uplift).  They didn't have a migration path from existing Linux applications -- there is a great one now.  They had average cluster management utilities and average software offerings -- now they can bundle their scheduler with moab, torque, interact with Suse Linux (Novell's offering), and will work with you to integrate your Linux existing configuration with their Cluster Server 2008 to provide a smooth transition as you learn how to use their tools and software.  The goal, of course, is world domination -- just like Linux!  The difference is that Microsoft has a lot of horsepower behind this effort.

I think they just might succeed.  Maybe I'll have to change religions one day.

Software Challenges in Oil and Gas

The Need

Geoscientists demand specialized “best of breed” software applications.  This has resulted in a proliferation of commercial software usage in oil and gas companies. While effective, the use of commercial software holds inherent strategic challenges.

 

Software Challenges

Software is an expensive but vital component of a successful oil and gas company’s E&P strategy. Due to the high cost of development and maintenance, organizations usually find it more cost-effective to purchase software than to develop it in-house.

When buying commercial software, each company must address the following challenges:

·                     How to sustain a competitive advantage if you use the same software as your competitors

·                     How to incorporate a needed new feature that the commercial applications do not support

·                     How to ensure that different data formats can be read by multiple applications.

Even when commercial software comprises the bulk of applications used in a company, researchers still need to create new programs.  If the expertise to produce and deploy proprietary software is missing, how does technology transfer from the research and development function to the operating units?

 

Extending Commercial Software

Extending commercial software with a Software Development Kit (SDK) offers one approach to resolving the challenges.  Many commercial applications have carved out a niche in a particular geoscience specialty such as seismic processing or petrophysics.  However, SDKs do have some drawbacks.

·                     Applications are often built using different software languages.

·                     Applications run on different operating systems.

·                     Licensing an application just to leverage its SDK can be expensive.

·                     Some SDKs are designed for use by professional developers and not by geoscientists.

·                     Adding new functionality into an existing application can be difficult.

·                     Software reuse among different SDKs is often impossible.

·                     Data exchange can be a problem.

Using a commercial SDK makes sense for some types of problems.  For instance, in the case of a seismic processing system, the problem can be expressed generically as the task of applying a sequence of operations to a collection of seismic traces on a cluster of machines.  If you are building a seismic processing module, then the odds are excellent that you can leverage a lot of the functionality in an SDK.

Many SDKs support plug-in extensions to a limited degree, but this does not mean that the application will be sufficiently extensible to support all needs. For example, multi-component seismic data is not supported in most of the current SDKs. One would have to add input/output functionality, visualization code and new data structures to handle multi-component data before effectively adding such new functionality to the software.

Data exchange issues are exacerbated by each SDK’s use of a different database.  Middleware technology (e.g. OpenSpirit, GeoShare) and workarounds involving file transfers can be employed in many cases, but sometimes developers must extend several applications with custom input and output code in order to complete a workflow for production use.

 

Developing Software Internally

In an era where the same software is used by most companies, it is tempting to try to gain a competitive advantage by developing more software in-house. However, the requirements for in-house software are just as demanding as those for commercial software, and development is handicapped by fewer resources and a relatively small user base that provides limited feedback.

Based on historical project data, internally developed software has a lifetime cost of approximately $100 for each line of code that is written.  This shows a clear need to avoid writing code whenever possible.  The easiest way to do this is to employ a “buy-versus-build” strategy and to focus on building only the most valuable new functionality.

Software developed internally needs to have a high perceived business value. This is achieved by building new functionality that cannot be purchased commercially, which occurs when the functionality is innovative or when it is a niche application with a small user base.

Oil companies frequently generate software requirements that are specific to business processes and the geological provinces being explored.  Commercial vendors cannot expect a reasonable return on investment if they tailor software to fit such specific business needs and so it is left to the oil and gas company to build its own software… or is it?

 

Open Source Strategy

The following quote is from the authors of Innovation Happens Elsewhere, Open Source as Business Strategyby Ron Goldman and Richard P. Gabriel (Morgan Kaufmann Publishers, San Francisco).

 “…companies need to find ways to use outside innovations and to become part of a distributed fabric of innovation through a combination of licensing and well chosen gifts. Although the concept of a gift may not at first seem to fit well with free-market capitalism, it might when thought of in the context of collaborating with others to build common commodity-like infrastructure.”

If the oil and gas company could make effective use of open source software, and give back to the open source project from time to time, then it could find itself with lower total cost of ownership, higher usability in its internal codes, and research software that finds solid usefulness in its operating units.  The goal of using an open source project is to avoid spending time writing code that others have already written – code that does the basic I/O, visualization, menuing, graphical user interface design, etc. and to concentrate on the software that gets the geophysics done.

 

Sustainable Open Source Development

Sustainable open source development requires a stable entity that can house the software, maintain the web presence, advertise and solicit for collaboration, and to own the intellectual property. The ideal entity is a non profit organization whose purpose is to foster collaboration and elevate the common good. It should have as one goal the education of the next generation of geoscientists.

Sustainable open source development also requires proper licensing that allows:

·                     businesses to use the product to make money,

·                     universities to use the product to house research projects and to train students,

·                     oil and gas companies to use the product to house proprietary programs.

 

In order for this kind of product development to be successful in the geophysical realm, it will need to:

·                     support multiple data formats,

·                     contain a standardized domain model,

·                     simplify development of new data visualization tools,

·                     encourage code reuse,

·                     have a user-friendly design,

·                     have the potential to evolve as a geoscience development platform,

·                     be built upon a solid foundation and framework.

 

Managing Open Source Software

Software is available.  For seismic processing, there are Madagascar, SU, CPSeis, and FreeUSP.  For quantitative interpretation there is QI and for general purpose interpretation utilities there is GeoCraft.  What we lack on some of these projects is the proper open-source license.  We also lack the non-profit organization that can bring geophysical open-source projects into an umbrella legal structure and merge the frameworks where appropriate.  We need the organization that can relicense with business-friendly licensing that helps the developers to justifiably reap some rewards for their efforts but that still allows the hobbyist and student to use the software.

 

Parting Thoughts

As a founding member of the GeoCraft team, I was able to think through many of these ideas, but the missing link then – and now – is the non-profit organization.  It is time to create that organization and to change the way the industry writes its software.  We should not want to damage the commercial software industry, but we do want to:

1. stop inefficient software development in the oil and gas companies,
2. create platforms for grad students to build their research upon, and
3. raise the bar for commercial offerings.

Open Source As a Foundation

We all, including your CIO, rely on open source.  If your company has outsourced its web presence (likely) then it has a very good chance of running on an Apache server on Linux.  So on the server side, open source IS the foundation.  Why is it so ubiquitous at this level but not at the enterprise desktop level?  There are many reasons, but at the core is lack of vision.  In 2002 I presented an open source concept for the enterprise desktop, but was not in the right place at the right time.  I present the vision for that model and ask your consideration.  Change is hard and established practices are etched in bedrock so this is a daunting proposal.

I challenge you to evaluate this concept (below) and try it for yourself.  Of course most of you have already done this on a personal level, but I'm talking about those of you who can influence your organization or company to try this at work.  Call it a pilot project.  Get $100,000 of internal play money (you know, the kind companies charge themselves in between departments), put a charge code in your SAP system (or whatever you are using), and try this out!

VISION

• A major company shows the world what a life on open source could look like.  It has capabilities for email, an office suite, web interfaces to its accounting system, HR system, knowledge sharing systems, etc.  It has an open source project management suite, software development suite, and has enough clout to nudge its major suppliers to ensure that their software can interoperate with open source solutions.  It shares documents and files with open source solutions enterprise-wide.  It deploys this solution worldwide, with tens of thousands of employees.
• The server farms at this corporation run an open source operating system, they run open source web servers, open source file sharing software,  open source database management services, and open source email services.
• The desktops run an open source windowing system.
• An open source virtual machine system allows individual desktops or server farms to present necessary proprietary operating systems for each user for specific applications that are not available in open source.
  
• CRITICAL COMPONENT: The corporation spends money on those open source solutions it uses.  It hires a team of software developers who work on the features that this company needs.  They write in those features, then give them back to the open source community.  On occasion, they write proprietary secret sauce software the company keeps inside itself -- but this is a rarity, not the norm.

PILOT PROJECT

1. One or two people (the future leaders of the software team above) put together a desktop suite that satisfies most needs of one department's employees.  They use virtual machine software (Commercial software is ok for the pilot!) to provide those necessary components that are provided commercially and can't be changed without major upheaval.
2. The department is switched over, starting with a small group.  Group by group switch each employee over to the new desktop.
3. Once the department is on the new desktop, fix each problem as it arises.  Make sure that everyone can do his necessary job functions with no interruption in service, no down time.  The team IS the help desk support for this project.
4. Run the department this way for six months. We are talking about 200 people or less for this pilot.  Everyone in the department must play, including the manager and the administrative assistants.
5. Write a financial analysis of the project.  Include support costs, switchover costs, software costs, maintenance costs.  Project the costs/benefits for five years.  Include the cost of the internal software team that will be giving back to the open source community.  This team is most likely your support team as well.  Contrast this analysis with a similar analysis of leaving things as they are currently established with proprietary systems.  Include costs of proprietary OS, proprietary email servers, proprietary office suite, etc.
6. Get permission to publish your results in CIO magazine.  You will turn some heads.

Good luck, and get to work!

Raising the Bar With Open Source Software

The geophysical community has spawned several open-source initiatives -- or maybe the open-source community has spawned several geophysical initiatives. Seismic Unix, Madagascar, CPSeis -- all for seismic processing! How do these help us besides providing students a sandbox in which to play and learn about the seismic method? They help in a couple of ways:

1. They provide the commodity processing capability at no charge
2. They provide a base-case for validation of other software

COMMODITY CAPABILITY

If you are willing to put up with the quirks of the software, you can put together a reasonable system with one of these packages, and set off to process data! You may not be the fastest kid on the block, but you can do the job. This capability raises the bar for commercial software vendors. No longer is "a complete system" good enough. The commercial system must have a compelling set of features that make you want to pay the fee. Raising the bar is a good thing for the geophysical community.

VALIDATION

Any student (or professional) working on a software algorithm now has two or three different packages to compare results with. Validation of the mathematics and the programming techniques are critical for making sure that you haven't made a sign error or a typo in your code. Of course, everything isn't available, but enough is there for many algorithms to be tested against.

More on this subject later!

Successful People

I read a great book yesterday, called "The Go-Giver" by Bob Burg and John David Mann.  It is a parable that teaches five principles for being successful in whatever you set out to do.  Here are the principles -- but you should read the book.  They teach by storytelling, which is one of the best ways to help you remember.

1. Your true worth is determined by how much more you give in value than you take in payment.
2. Your income is determined by how many people you serve and how well you serve them.
3. Your influence is determined by how abundantly you place other people's interests first.
4. The most valuable gift you have to offer is yourself.
5. The key to successful giving is to stay open to receiving.

How much of yourself are you giving away?  What do you do that puts others' interests before your own?

Think of how much better your driving experience would be if everyone decided that it would be ok for someone to go in front of them on the road.  I might even enjoy driving in Houston again!