CIO woes

Being the CIO is quite a bit different than being the HPC manager. A couple of months ago I talked about doing a pilot project with an open-source office suite, open-source email. Well, now I'm on the other side of that discussion, and it is especially important since I am in a small (i.e. not multi-billion dollar) corporation. We have about half the company using OpenOffice and Thunderbird/Lightning for email/calendar. The entire company isn't on one solution, however, so that presents its own challenges. With users on three mail clients (some use Outlook, some use Evolution, and some use T-bird) someone in the IT department needs to keep up with all of that software. Some people use OpenOffice, some use Microsoft Office. Some are only provided with a Linux workstation, others have only a Windows box. Some are on Office 2007, others on Office 2003.

Cleaning house is never fun, but now it's my house and I do need to streamline support costs and make things more efficient. I believe we can get there with a good open source suite. More on that as things unfold.

Speaking of open-source, I read a good book several years ago at the urging of Duncan Child, one of my previous associates, called "Dreaming in Code". It walks through product development of Chandler, a "personal productivity assistant". Version 1.0 is now available, so I downloaded it for my laptop. (Hey, I'm the IT manager, I can do whatever I want.) It looks like it might be useful, but will require more off-line experimenting. Unfortunately, there is no time during working hours to do anything but work. Not a bad thing, just a comment. Well, the clock is ticking, back to work.

Have a great day!

Bill

Proposal for HPC Association

Proposal to Create an HPC Association

EXECUTIVE SUMMARY

Based on the May 5th HPC Forum meeting and subsequent meetings, I am modifying my earlier proposal for the establishment of a non-profit association for HPC users in the Energy sector to include all HPC users.  This association will be driven by the needs all parties who are interested in HPC.  As part of the larger group, an Energy Special Interest Group could be formed.  For the economies of scale, it seems best to form a more inclusive association that would include Aerospace, Medical, Financials, Pharma, Automotive, Academia and others.  As a non-profit, the association will be able to create and fund projects for the various industries as desired by the membership or by sub-groups within the membership.

BACKGROUND

After discussions with several companies and individuals subsequent to the March 5th HPC Workshop at Rice University, it became clear that a need existed for some kind of HPC group for the Oil and Gas industry where users of the technologies could collaborate, share ideas, discuss issues, and propose projects of mutual interest.  On May 5th a small team launched an HPC Forum, inviting only Oil company HPC personnel to discuss this concept.  The idea was well received and will continue as a monthly or bi-monthly meeting but will be expanded to include other users of HPC technology in the industry.

 

On May 6th another meeting was hosted by an individual with a research proposal for an open-source project involving seismic data processing in the HPC arena.  As a result of this meeting, it became clear that a mechanism for funding projects such as this one was lacking.

WHY A NON-PROFIT

The need for an association and for methods to conduct research in HPC have been identified.  A non-profit corporation provides an effective mechanism for managing such a group or association.  A non-profit status facilitates collaboration with universities and government entities.  The non-profit can manage research projects in the HPC area for its members and can maintain varying levels of participation and disclosure for those projects.  Because of non-profit status, participants can join projects with the certainty that no large company will be able to purchase the organization to gain access to any intellectual property created.  As the organization matures, it can expand to assist universities with HPC programs by providing scholarships and grants, and by funding research projects at the universities.

NEXT STEPS

 

At the next HPC Forum on July 14, 2009 I would like to discuss this idea.  We need to put together the mission statement for the HPC Forum and decide if a non-profit is agreeable as a mechanism for sustained activity in the HPC-Energy area.  If we decide to move forward, we will need to create the organization, agree on an initial board of directors, and file 501.C.3 paperwork with the US government and incorporation papers with the State of Texas.

 

PROPOSED NAME

The HPC Association (HPCA)

 

PROPOSED VISION STATEMENT

 

The HPCA provides its members with a collaborative society consisting of individuals, corporations, and government entities who are associated with High Performance Computing (HPC) in various capacities.  HPCA facilitates research and development of new technologies in HPC and promotes academic HPC programs.  HPCA strives to provide members with resources and knowledge to better solve the complex emerging computational problems that face the world.

 

PROPOSED DESCRIPTION

 

The HPCA is a non-profit corporation.  It  consists of a collaborative society that includes everyone associated with High Performance Computing (HPC).  This includes HPC users such as academics, scientists, computer systems personnel, government research laboratories, and government agencies, as well as HPC providers such as software and hardware suppliers, consultants, and niche technology companies.  It is open to all interested parties.  The association collects dues, provides processes to distribute knowledge, allows for advertising venues, and works with symposia and conventions to help bring the HPC community together.

PROPOSED MISSION STATEMENT

 

The HPC Association is established to promote best practices and research in the area of high performance computing (HPC).  It serves the common good by promoting HPC for a variety of industries, increasing knowledge, and facilitating research into HPC methods.  It provides a forum for discussion and collaboration on all aspects of HPC including usage, operations, hardware, and software.  The Association provides funding for HPC research and development and strives to assist universities with HPC programs.

 

HPC ENERGY FORUM

 

The continuance of the HPC Energy Forum is encouraged as a special interest group within the HPC Association.  This forum will include the users and providers of HPC technology in the energy industry and includes universities, service companies, oil and gas companies, and HPC solutions providers.  It will solicit sponsorship of the forum from various providers of HPC products or services.  Participation at the forum will be open to HPC users, HPC systems and operations personnel, managers of HPC operations, software engineers, service providers, and solutions providers.

 

“Logos and Egos are Checked at the Door”

Bill Menger              Page 2              7/13/2009

The Steady Advance of Linux

I don't remember the year, but I'm thinking it was 1992 when I downloaded 18 floppy disks' worth of the tamu-0.99 version of Linux and put it on my home 386. It was one tar image, and I recall disk #15 having a bad sector so I had to re-download it. I believe the modem was about 14.4Kbaud, so we're talking about serious time spent downloading. I was too cheap to buy Debian, which I believe was just out.

Amazingly, I was able to put this to work at work! I had been using SCO Xenix on a machine that was my Ingres database server and development machine, but the ethernet package that came with Xenix (for an extra fee) didn't work very well. I put the tamu distro on that machine on another disk and rebooted it, and the networking was good! With a little work, Ingres worked on the old tamu distribution and I was back running again, but this time on an unsupported O/S that ran better than Xenix.

Fast forward to last week, when I downloaded Ubuntu 9.04 ISO image and burned a CD. I put it on a dual Nehalem Supermicro machine with an Nvidia Quadro 4500 graphics card. Now it is 64 bit, came with X-windows, automatically connected to time servers so my clock is always in sync, has a default browser, office suite, email client, and development tools. It automatically detects when upgrades are needed to packages. With Cat 5e cables in the house it talks internally at 100Mbit (because I'm still cheap and won't buy a gigabit switch), and connects to the outside world at over 2Mbits/second. Granted, a few years have elapsed, but I'm still amazed that I could hook up this much power to the internet in about two hours, including my download time for the boot disk image.

Is my family ready for Linux? Perhaps. That was not the case in 1992 when I was told to quit messing up the family computer, but now the question I get is "If you put that in the kitchen can I get to my email?" with an implied green light as long as I can provide the basic functionality. Is my business ready for Linux? Definitely. I need to keep overhead low, and providing services in the HPC and software world means that I need to have Linux available anyway... so why shouldn't it run my office applications as well? What about Microsoft? They have a dominant place, and still run the O/S at my house for video editing, calendaring, Turbotax, etc. We also use the Office 2003 suite. But what is significant is that we have a choice. My Google calendar syncs with Outlook (from Microsoft), and my OpenOffice 3.1 allows me to modify my Office 2003 documents into ".pdf's" with incredible ease. If I want to work on the Linux machine I can create, edit, and print a document that is compatible with the Vista-hosted Microsoft Word program.

Life is good. Linux has come of age.

SiCortex

Something strange is happening at SiCortex.  A friend says the bank has taken control and laid off the staff.  I hope they aren't forced into bankruptcy -- it would be very bad for HPC.  Their machine is unique in its low-power approach to supercomputing... kind of like an IBM Blue-Gene but with lots of memory per core, and a 500Gbit/second network on top of that.

Keep your ears open, probably an announcement today or tomorrow.

And I was on the last step of installing CPSeis on their Houston machine.  It was SO EASY!  Fribbits.

Microsoft's Cluster Server 2008

I listened in on a webinar last week hosted by RAI and Microsoft.  Being a Linux fan for many years, I was skeptical of what the new Microsoft Cluster server could do -- especially since my team had evaluated it a couple of years ago and thought "ho-hum".  I will say they really surprised me with the speed at which they have taken this "also-ran" operating system and made it into a viable challenge for the Linux geek community to seriously consider.

If I were managing a new HPC operation I would give it serious consideration.  They had an issue with the networking interconnect speeds -- it's fixed (30% uplift).  They didn't have a migration path from existing Linux applications -- there is a great one now.  They had average cluster management utilities and average software offerings -- now they can bundle their scheduler with moab, torque, interact with Suse Linux (Novell's offering), and will work with you to integrate your Linux existing configuration with their Cluster Server 2008 to provide a smooth transition as you learn how to use their tools and software.  The goal, of course, is world domination -- just like Linux!  The difference is that Microsoft has a lot of horsepower behind this effort.

I think they just might succeed.  Maybe I'll have to change religions one day.

Software Challenges in Oil and Gas

The Need

Geoscientists demand specialized “best of breed” software applications.  This has resulted in a proliferation of commercial software usage in oil and gas companies. While effective, the use of commercial software holds inherent strategic challenges.

 

Software Challenges

Software is an expensive but vital component of a successful oil and gas company’s E&P strategy. Due to the high cost of development and maintenance, organizations usually find it more cost-effective to purchase software than to develop it in-house.

When buying commercial software, each company must address the following challenges:

·                     How to sustain a competitive advantage if you use the same software as your competitors

·                     How to incorporate a needed new feature that the commercial applications do not support

·                     How to ensure that different data formats can be read by multiple applications.

Even when commercial software comprises the bulk of applications used in a company, researchers still need to create new programs.  If the expertise to produce and deploy proprietary software is missing, how does technology transfer from the research and development function to the operating units?

 

Extending Commercial Software

Extending commercial software with a Software Development Kit (SDK) offers one approach to resolving the challenges.  Many commercial applications have carved out a niche in a particular geoscience specialty such as seismic processing or petrophysics.  However, SDKs do have some drawbacks.

·                     Applications are often built using different software languages.

·                     Applications run on different operating systems.

·                     Licensing an application just to leverage its SDK can be expensive.

·                     Some SDKs are designed for use by professional developers and not by geoscientists.

·                     Adding new functionality into an existing application can be difficult.

·                     Software reuse among different SDKs is often impossible.

·                     Data exchange can be a problem.

Using a commercial SDK makes sense for some types of problems.  For instance, in the case of a seismic processing system, the problem can be expressed generically as the task of applying a sequence of operations to a collection of seismic traces on a cluster of machines.  If you are building a seismic processing module, then the odds are excellent that you can leverage a lot of the functionality in an SDK.

Many SDKs support plug-in extensions to a limited degree, but this does not mean that the application will be sufficiently extensible to support all needs. For example, multi-component seismic data is not supported in most of the current SDKs. One would have to add input/output functionality, visualization code and new data structures to handle multi-component data before effectively adding such new functionality to the software.

Data exchange issues are exacerbated by each SDK’s use of a different database.  Middleware technology (e.g. OpenSpirit, GeoShare) and workarounds involving file transfers can be employed in many cases, but sometimes developers must extend several applications with custom input and output code in order to complete a workflow for production use.

 

Developing Software Internally

In an era where the same software is used by most companies, it is tempting to try to gain a competitive advantage by developing more software in-house. However, the requirements for in-house software are just as demanding as those for commercial software, and development is handicapped by fewer resources and a relatively small user base that provides limited feedback.

Based on historical project data, internally developed software has a lifetime cost of approximately $100 for each line of code that is written.  This shows a clear need to avoid writing code whenever possible.  The easiest way to do this is to employ a “buy-versus-build” strategy and to focus on building only the most valuable new functionality.

Software developed internally needs to have a high perceived business value. This is achieved by building new functionality that cannot be purchased commercially, which occurs when the functionality is innovative or when it is a niche application with a small user base.

Oil companies frequently generate software requirements that are specific to business processes and the geological provinces being explored.  Commercial vendors cannot expect a reasonable return on investment if they tailor software to fit such specific business needs and so it is left to the oil and gas company to build its own software… or is it?

 

Open Source Strategy

The following quote is from the authors of Innovation Happens Elsewhere, Open Source as Business Strategyby Ron Goldman and Richard P. Gabriel (Morgan Kaufmann Publishers, San Francisco).

 “…companies need to find ways to use outside innovations and to become part of a distributed fabric of innovation through a combination of licensing and well chosen gifts. Although the concept of a gift may not at first seem to fit well with free-market capitalism, it might when thought of in the context of collaborating with others to build common commodity-like infrastructure.”

If the oil and gas company could make effective use of open source software, and give back to the open source project from time to time, then it could find itself with lower total cost of ownership, higher usability in its internal codes, and research software that finds solid usefulness in its operating units.  The goal of using an open source project is to avoid spending time writing code that others have already written – code that does the basic I/O, visualization, menuing, graphical user interface design, etc. and to concentrate on the software that gets the geophysics done.

 

Sustainable Open Source Development

Sustainable open source development requires a stable entity that can house the software, maintain the web presence, advertise and solicit for collaboration, and to own the intellectual property. The ideal entity is a non profit organization whose purpose is to foster collaboration and elevate the common good. It should have as one goal the education of the next generation of geoscientists.

Sustainable open source development also requires proper licensing that allows:

·                     businesses to use the product to make money,

·                     universities to use the product to house research projects and to train students,

·                     oil and gas companies to use the product to house proprietary programs.

 

In order for this kind of product development to be successful in the geophysical realm, it will need to:

·                     support multiple data formats,

·                     contain a standardized domain model,

·                     simplify development of new data visualization tools,

·                     encourage code reuse,

·                     have a user-friendly design,

·                     have the potential to evolve as a geoscience development platform,

·                     be built upon a solid foundation and framework.

 

Managing Open Source Software

Software is available.  For seismic processing, there are Madagascar, SU, CPSeis, and FreeUSP.  For quantitative interpretation there is QI and for general purpose interpretation utilities there is GeoCraft.  What we lack on some of these projects is the proper open-source license.  We also lack the non-profit organization that can bring geophysical open-source projects into an umbrella legal structure and merge the frameworks where appropriate.  We need the organization that can relicense with business-friendly licensing that helps the developers to justifiably reap some rewards for their efforts but that still allows the hobbyist and student to use the software.

 

Parting Thoughts

As a founding member of the GeoCraft team, I was able to think through many of these ideas, but the missing link then – and now – is the non-profit organization.  It is time to create that organization and to change the way the industry writes its software.  We should not want to damage the commercial software industry, but we do want to:

1. stop inefficient software development in the oil and gas companies,
2. create platforms for grad students to build their research upon, and
3. raise the bar for commercial offerings.

Open Source As a Foundation

We all, including your CIO, rely on open source.  If your company has outsourced its web presence (likely) then it has a very good chance of running on an Apache server on Linux.  So on the server side, open source IS the foundation.  Why is it so ubiquitous at this level but not at the enterprise desktop level?  There are many reasons, but at the core is lack of vision.  In 2002 I presented an open source concept for the enterprise desktop, but was not in the right place at the right time.  I present the vision for that model and ask your consideration.  Change is hard and established practices are etched in bedrock so this is a daunting proposal.

I challenge you to evaluate this concept (below) and try it for yourself.  Of course most of you have already done this on a personal level, but I'm talking about those of you who can influence your organization or company to try this at work.  Call it a pilot project.  Get $100,000 of internal play money (you know, the kind companies charge themselves in between departments), put a charge code in your SAP system (or whatever you are using), and try this out!

VISION

• A major company shows the world what a life on open source could look like.  It has capabilities for email, an office suite, web interfaces to its accounting system, HR system, knowledge sharing systems, etc.  It has an open source project management suite, software development suite, and has enough clout to nudge its major suppliers to ensure that their software can interoperate with open source solutions.  It shares documents and files with open source solutions enterprise-wide.  It deploys this solution worldwide, with tens of thousands of employees.
• The server farms at this corporation run an open source operating system, they run open source web servers, open source file sharing software,  open source database management services, and open source email services.
• The desktops run an open source windowing system.
• An open source virtual machine system allows individual desktops or server farms to present necessary proprietary operating systems for each user for specific applications that are not available in open source.
  
• CRITICAL COMPONENT: The corporation spends money on those open source solutions it uses.  It hires a team of software developers who work on the features that this company needs.  They write in those features, then give them back to the open source community.  On occasion, they write proprietary secret sauce software the company keeps inside itself -- but this is a rarity, not the norm.

PILOT PROJECT

1. One or two people (the future leaders of the software team above) put together a desktop suite that satisfies most needs of one department's employees.  They use virtual machine software (Commercial software is ok for the pilot!) to provide those necessary components that are provided commercially and can't be changed without major upheaval.
2. The department is switched over, starting with a small group.  Group by group switch each employee over to the new desktop.
3. Once the department is on the new desktop, fix each problem as it arises.  Make sure that everyone can do his necessary job functions with no interruption in service, no down time.  The team IS the help desk support for this project.
4. Run the department this way for six months. We are talking about 200 people or less for this pilot.  Everyone in the department must play, including the manager and the administrative assistants.
5. Write a financial analysis of the project.  Include support costs, switchover costs, software costs, maintenance costs.  Project the costs/benefits for five years.  Include the cost of the internal software team that will be giving back to the open source community.  This team is most likely your support team as well.  Contrast this analysis with a similar analysis of leaving things as they are currently established with proprietary systems.  Include costs of proprietary OS, proprietary email servers, proprietary office suite, etc.
6. Get permission to publish your results in CIO magazine.  You will turn some heads.

Good luck, and get to work!