John Wang online

Base58: Fast Hashing with GMP

John Wang — Fri, 17 Feb 2012 19:37:42 +0000

Base58 is an alternative to Base64 that is growing in popularity for case-sensitive encodings due to several characteristics including multi-protocol-safety and human-readability. It is used by Flickr, Bitcoin, and now Grokbase. The general characteristics of Base58 are that it uses the protocol-safe alpha-numeric alphabet (Base62) and excludes easy to confuse digits. Both Flickr and Grokbase use [0-9a-zA-Z] excluding [0O1l] while Bitcoin uses [0-9A-Za-z] excluding [0O1l].

Grokbase uses MD5 hashes of email addresses for UIDs and when I did the update, there were over 500,000 user profiles in the system so I wanted a fast encoder. I ended using GMP with tr and released the code as Encode::Base58::GMP for Perl and base58gmp for Ruby.

As for how much faster, I ran a few benchmarks comparing Encode::Base58::GMP to the pure Perl Encode::Base58 implementation by Miyagawa and found that the GMP version was about 700x (70,000%) faster encoding MD5 hex values using the Ubuntu 11.10 system Perl. However, for 32-bit integers, the pure Perl version was faster by about 50%.

Two notes about these numbers:

Ubuntu 11.10 x86_64 system Perl isn’t compiled with 64-bit integer support and Encode::Base58 needs integers so I used bignum to enable hex() support. One possibilty is to compile a Perl with 64-bit int support; however, Ubuntu system Perl is part of my stack and many others also rely on system Perl so I left it as that.
Math::GMPz can accept hex values natively and since the goal was to generate MD5 values, I bypassed the integer conversion used with Encode::Base58. When adding integer conversion, GMP based encoding is still faster but drops by an order of magnitude.

70,000% Faster for MD5 (BigInts)

use strict;
use warnings;

use Benchmark qw/cmpthese/;
use bignum qw/hex/;
use Encode::Base58;
use Encode::Base58::GMP;
use Digest::MD5 qw/md5_hex/;

cmpthese(-5, {
  pp  => sub {
    Encode::Base58::encode_base58(hex(md5_hex(rand())))
  },
  gmp => sub {
    Encode::Base58::GMP::encode_base58('0x'.md5_hex(rand()))
  }
} );

Here is a sample result:

       Rate     pp    gmp
pp    111/s     --  -100%
gmp 79644/s 71484%     --

33% Slower on 32-bit Ints

Here are some results using 32-bit integers.

cmpthese(-5, {
  pp  => sub {
    Encode::Base58::encode_base58(int(rand 2**32))
  },
  gmp => sub {
    Encode::Base58::GMP::encode_base58(int(rand 2**32))
  }
} );

       Rate  gmp   pp
gmp 66733/s   -- -32%
pp  98803/s  48%   --

Conclusion

While GMP-based encoding is about 33% slower for 32-bit ints, it is significantly faster for 64-bit ints on Ubuntu 11.10 system Perl and very desirable for creating hashes. Numbers will vary with a Perl compiled with 64-bit int support.

Note: Grokbase User URLs

Currently, the Grokbase profile URLs use unsalted MD5 Base58 hashes of email addresses so you can find anyone’s profile page using the following:

use Encode::Base58::GMP qw/md5_base58/;

my $url = 'http://grokbase.com/user/-/'
        . md5_base58( 'test@example.com' );

The ‘-’ after ‘/user’ will get redirected to the user’s display name as shown in their emails.

Using an unsalted MD5 hash is similar to how Gravatar works and has the added benefit of simply requiring a hex to Base58 conversion to convert a Gravatar UID to a Grokbase UID.

Credits:

Marcus Bointon‘s article on Base62 encoding using GMP provided the initial idea of using GMP for arbitrary-base transcoding.
Sam Rawlins provided an update for Ruby’s gmp to handle arbitrary-base decoding.
Case Van Horsen provided an update for Python’s gmpy to support Base37-62.

Confident Code Using Systematic, Automated Input Validation

John Wang — Fri, 11 Nov 2011 16:50:16 +0000

The other day on Hacker News, there was a good discussion of Avdi Grimm‘s talk on how to write Confident Code given at this year’s Ruby Midwest. He covers four areas: gather input, perform work, deliver results and handle failure. Here are links to the code example showing the initial, timid and final, confident code. I’ll just cover the input gathering step because I think systematic, automated, input validation it is vital for improving coding efficiency and minimizing boilerplate.

Most of this can actually be done using something like Ruby’s Doodle. It does not appear that Doodle is very popular yet (21 all time messages in the mailing list) though I think it would be good if it were more so.

More popular is Stevan Little‘s Moose for Perl which has 28 named contributors. While I like to read about many technologies I generally like to use more mature and popular technologies than less so I’ve been using Perl’s Moose and haven’t coded with Doodle yet.

To show how Moose is powerful, let’s write the Confident Code example in Perl

Confident Code in Moose

I whipped this up for discussion so it can definitely be refined, however, it should get the ideas across. Here are some things that are covered:

Input Validation: Inputs are typed and given default values in both attributes (e.g. width as an Integer with default 40) and methods (e.g. messages as an ArrayRef of Strings – ArrayRef[Str])
Separation of Concerns: attributes and methods are conveniently separated for easier maintenance and testing
Output: IO::All is used for output, defaulting to STDOUT (i.e. ‘-’)

use MooseX::Declare;

class CowsayTheMooseway {
  use Capture::Tiny qw(capture_merged);
  use IO::All;
  use Log::Log4perl qw(:easy);
  Log::Log4perl->easy_init($INFO);

  has width       => ( isa => 'Int', is => 'rw', default => 40      );
  has eyes        => ( isa => 'Str', is => 'rw', default => ''      );
  has cowfile     => ( isa => 'Str', is => 'rw', default => 'moose' );
  has destination => ( isa => 'Str', is => 'rw', default => '-'     );

  method command {
    my $command = 'cowsay -W ' . $self->width;
    $command .= ' -e '.$self->eyes    if $self->eyes;
    $command .= ' -f '.$self->cowfile if $self->cowfile;
  }

  method say (ArrayRef[Str] $messages) {
    my $command = $self->command;
    my $output = capture_merged {
      system "$command $_" for @$messages;
    };
    $output > io $self->destination;
    INFO "wrote to ".$self->destination;
  }
}

Note how the input attributes and methods are all separated cleanly with their own input types.

Here’s how to run the class:

use CowsayTheMooseway;

CowsayTheMooseway->new(
  destination => 'moo.txt'
)->say([
  'Psst...Want More Confidence?',
  'Here is How',
  'Moose This Way'
]);

And the results:

 ______________________________
< Psst...Want More Confidence? >
 ------------------------------
  \
   \   \_\_    _/_/
    \      \__/
           (oo)\_______
           (__)\       )\/\
               ||----w |
               ||     ||
 _____________
< Here is How >
 -------------
  \
   \   \_\_    _/_/
    \      \__/
           (oo)\_______
           (__)\       )\/\
               ||----w |
               ||     ||
 ________________
< Moose This Way >
 ----------------
  \
   \   \_\_    _/_/
    \      \__/
           (oo)\_______
           (__)\       )\/\
               ||----w |
               ||     ||

Let me know what you think. Should this type of coding be done in Ruby? Is Doodle the way to go or is there something else? Can the Perl code be improved?

Explaining Why Moose is Awesome – The Direct Way

John Wang — Tue, 01 Nov 2011 16:43:11 +0000

The Perl community knows that Moose is awesome as evidenced by the amount of talks at YAPC and the number of times it is mentioned on blog articles and discussion boards. However, to me, what often seems to get lost is why Moose is awesome and why it is relevant, especially to those outside the Perl community. At the same time, I think it is vitally important to simply and directly convey why Moose is awesome to both increase Perl awareness and encourage other languages to adopt similar techniques.

This article covers how I explain Moose and why I think it not only addresses the key concerns with Perl but makes it more attractive when compared to some other languages as they are today.

The ideas here started when I was recently showing off a project of mine to a YC-funded Xoogler that asked me what it was written in. When I told him it was mostly Perl, he said, “Oh Perl, you’re old school!” I could have left it at that if it were true, but it wasn’t and I wanted to explain why it wasn’t. I had specifically chosen Perl for this project in 2011 over Python because it provided things that were not yet in (or popular in) other languages. This is how I presented it.

Why: Moose makes Perl more Maintainable than Other Languages

Nothing is more attention grabbing than saying something opposite to what people expect so this is exactly what I said. Of course, the gauntlet was already thrown so I wanted to respond in kind. I was able to say this honestly because I backed it up using my recent experiences using Perl and JavaScript with some Ruby, Python and C#. At the same time, it is important to tell it as it is and not sugar coat things. I’ve had my share of head scratching when looking at my own 6-month old pre-Moose Perl code so I’m painfully aware of Perl’s write once reputation.

The way I explain it is that while it can be argued that some languages may be cleaner than Perl on a line-by-line basis, my view is that Moose makes Perl cleaner and more maintainable on a class and method basis. By providing an expressive attribute getter/setter system, Moose helps you freeze APIs and limit the scope for maintainability to the attribute / method level. It does this by adding one key feature…

How: Moose adds Static Typing to Getters and Setters

What Moose really does is add type checking to getters and setters. While it supports some built-in types (integers, numbers, strings, arrayrefs, hashrefs, etc.), the real power is in composite types like an arrayref of objects e.g. ‘ArrayRef[Email::Simple]‘. By freezing the API, Moose improves coding speed and reduces errors, enabling me to write 1000s of lines of code and having it all run with minimal typo correction. The great thing about this explanation is that static typing is something that people are generally looking at and interested in for many dynamic languages so once you get this point across, the benefits are easily understood and don’t have to be explained further.

Bonus: Moose Eliminates Write Once Code

Generally, I don’t have to go further than explaining static typing with Moose; however, it can be useful to note that Moose also eliminates write-once Perl code by providing a “template” for getter/setter creation. Creating classes this way ensures that your classes will follow well-ordered, well-understood convention. After having to maintain several old pre-Moose Perl apps, I was pleasantly surprised when I opened a 6 month old Moose class and had to double-check to see if it was written months ago or just the other day.

Summary

So there you have it. In one fell swoop, the key benefits of Moose are explained in a way that addresses long-running, key concerns with Perl with key features not yet readily available, but often desired, in other languages.

By succinctly explaining why Moose is awesome, hopefully we can raise more awareness for the great work being done by the Moose team and the Perl ecosystem.

What You Can Do

If you like this approach, one thing that can be done to more quickly explain the effectiveness of Moose programming is to create side-by-side code examples of code with Moose and without Moose, in other languages. This is often done to compare and contrast languages and I think it would be useful to more succinctly explain the benefits here. Expressive modern Perl code can impress those outside the community, there just needs to be more of it.

Perl and Python – Two Nights in Silicon Valley

John Wang — Mon, 31 Oct 2011 07:22:34 +0000

One of the well-touted advantages of living in Silicon Valley is that it is a crucible of talent. Along with colleagues, conferences and unconferences, there are many regular user groups in the valley, of which I’ve attended several. This past week, I decided to check out the local SF Perl Mongers and BayPIGgies (Python) which both had meetings.

Night One – SF Perl Mongers

I’ve been to two YAPCs but this was my first SF Mongers meeting and I was pleasantly surprised. As it happens, the talk was being given by Ingy döt Net, the creator of YAML, who I met briefly several years ago. I also got to meet up with Larry Wall and Miyagawa, both of whom I last spoke with at YAPC::NA, as well as new acquaintances including Catalyst core team members Tom Doran (who I recognized from his Gravatar) and Florian Ragwitz (rafl). Tom and Florian were here from Europe courtesy of Google for their participation in the Google Summer of Code.

Ingy’s talk was on Acmeism, his movement to simultanenously write code in multiple object-oriented languages including JavaScript, Perl, PHP, Python and Ruby among many others. The genesis of this project seemed to be from his experience with YAML where each language would have it’s own, often slightly different, implementation. With Acemeism, changes could be pushed out to multiple languages with one code update. I can see the advantage of Acmeism for a cross-language library that requires interoperability. For example, it would be great if there was only one interoperable version of LZF compression; however, it seems that not only does each language have a non-interoperable implementation, there may be multiple non-interoperable implementations per language. More generally, I’ve been interested in the compilation of interpreted code to C/C++ for the performance boost, a la Facebook’s Hip Hop, and did mentioned this to Larry in the context of Moose and Perl 6.

Afterwards, we went over to 21st Amendment Brewery for some drinks and dinner.

Night Two – Python BayPIGgies

This meeting was run by a few folks from Google, including JJ Behrens, from the YouTube API team, and Wesley Chun, a developer advocate based in SF. Both JJ and Wesley were held up so we were treated to a talk by Alex Martelli the #2 ranked “Famous Python Programmer at Google” according to Wesley’s slides he was giving. Alex, JJ and Wesley all provided great talks and insights.

Some good takeaways from the night included:

The original spidering code for the crawler was done in Python by a part-timer who received a 5% equity stake.
Back when Google Video and YouTube were competitors, the YouTube team was out-innovating the Google team because they were using Python while the Google team was using C++.
While Python is one of the approved deployment languages, it is used mostly in the back-end sysadmin, deployment and QA code. The user apps are either in C++ or are moving that way which can provide up to a 20% speed up.
PyPy was recommended by Wesley as a faster version of Python that is also faster than CPython on many tests. Digging into PyPy a bit afterwards, I was pleasantly surprised to learn it’s written in RPython, a statically-typed restricted version.

From Google’s perspective, it seems like Python is great for rapid prototyping and competitive feature deployment as well as back-end work; however, once the feature set is relatively stable, there are performance gains to be had from porting the user-facing applications. While perhaps not the approach, everyone would take, it’s interesting to hear their perspective.

Afterwards we broke up and went our own ways. There did not seem to be the same desire to hang out afterwards as in SF.

Both presentations were posted online:

Wesley’s Python @ Google (IO 2011) presentation

JJ’s YouTube for Business presentation

Summary

I was impressed by the people and knowledge gained at both meetings. They were intimate in size so it was easy to get to know people and engage in conversation. I’m looking forward to more meetings in the future.

Filtering Popular Mailing List Discussions with DateTime

John Wang — Tue, 11 Oct 2011 14:56:38 +0000

Discovering Information via Better Browsing

A great way to get into the thick of a new software project you want to learn or need to use is to check out popular mailing list discussions. Typically these will have detailed answers and multiple points of view. Recognizing this, Grokbase was designed to sort many things by popularity, or activity volume, such as discussions, groups, users, etc. I found it useful enough that I made it the default monthly view and especially like getting a quick overview of a group by browsing these pages on my iPad using the monthly prev and next links.

I was recently chatting with Dave Rolsky, creator of the stalwart DateTime and newer Courriel email distributions, who mentioned it would be great if you could select popular discussions based on time periods longer than a month, such as 6 months or a year. It was easy to see where he was going with this. While sorting across all time is useful for curiosity once in a while, more recent popular discussions are what’s needed to keep up to date,

Needing a break from a larger project, I decided to see how quickly I could implement this. In the end, it took just a few hours from start to deployment, mostly helped by the fact that DateTime performs excellent date gymnastics. Finding the correct date to use was the easiest part using DateTime’s subtraction feature. DateTime’s functionality and excellent documentation made adding this feature a snap. In fact, the only thing it cost me was a little sleep

You can see the results on any of the more popular mailing lists such as Perl5 Porters (P5P) as shown here. Under the popular tab, there are now filters for time periods from 3 months to 5 years and for all time. I have to say I find it fun to dig into mailing lists this way. More than blogs or articles, diving into P5P really makes you feel like you’re at a YAPC.

For more on DateTime, checkout the MetaCPAN page and the Grokbase list archive.

Right now, this only accessible on the top popular tab. It is also enabled only for more popular groups where this is more likely to be useful. Try it out and let me know what you think.

A Tribute to Steve Jobs (1955-2011)

John Wang — Thu, 06 Oct 2011 14:58:06 +0000

Thinking Differently

Steve Jobs has passed away and the world has lost someone that showed what could be accomplished with a singular and distinctive vision. His was an approach that fused usability, appearance and functionality into a cohesive whole. I’ve read many of the tributes and articles and two of which that stood out from me were ons that showed him on a more personal level as told by Stephen Wolfram and Al Alcorn.

My condolences to his family, friends, and colleagues during this difficult time.

For my tribute, I’ll simply list a few of his quotations that helped focus his efforts on his and his team’s achievements.

Inspiration
Design
Marketing
Recruiting
Managing
Revenues
Discipline
Follow Your Heart

1. Inspiration

We think the Mac will sell zillions, but we didn’t build the Mac for anybody else. We built it for ourselves. We were the group of people who were going to judge whether it was great or not. We weren’t going to go out and do market research. We just wanted to build the best thing we could build.
Innovation has nothing to do with how many R&D dollars you have. When Apple came up with the Mac, IBM was spending at least 100 times more on R&D. It’s not about money. It’s about the people you have, how you’re led, and how much you get it.

2. Design

Unfortunately, that’s too rare a commodity. A lot of people in our industry haven’t had very diverse experiences. So they don’t have enough dots to connect, and they end up with very linear solutions without a broad perspective on the problem. The broader one’s understanding of the human experience, the better design we will have.
That’s been one of my mantras — focus and simplicity. Simple can be harder than complex: You have to work hard to get your thinking clean to make it simple. But it’s worth it in the end because once you get there, you can move mountains.
Design is a funny word. Some people think design means how it looks. But of course, if you dig deeper, it’s really how it works. The design of the Mac wasn’t what it looked like, although that was part of it. Primarily, it was how it worked. To design something really well, you have to get it. You have to really grok what it’s all about. It takes a passionate commitment to really thoroughly understand something, chew it up, not just quickly swallow it. Most people don’t take the time to do that.

3. Marketing

We’ve never worried about numbers. In the market place, Apple is trying to focus the spotlight on products, because products really make a difference. [...] Ad campaigns are necessary for competition; IBM’s ads are everywhere. But good PR educates people; that’s all it is. You can’t con people in this business. The products speak for themselves.
It’s not about pop culture, and it’s not about fooling people, and it’s not about convincing people that they want something they don’t. We figure out what we want. And I think we’re pretty good at having the right discipline to think through whether a lot of other people are going to want it, too. That’s what we get paid to do. We just want to make great products.
Our DNA is as a consumer company – for that individual customer who’s voting thumbs up or thumbs down. That’s who we think about. And we think that our job is to take responsibility for the complete user experience. And if it’s not up to par, it’s our fault, plain and simply.

4. Recruiting

Recruiting is hard. It’s just finding the needles in the haystack. You can’t know enough in a one-hour interview. So, in the end, it’s ultimately based on your gut. How do I feel about this person? What are they like when they’re challenged? I ask everybody that: ‘Why are you here?’ The answers themselves are not what you’re looking for. It’s the meta-data
When I hire somebody really senior, competence is the ante. They have to be really smart. But the real issue for me is, Are they going to fall in love with Apple? Because if they fall in love with Apple, everything else will take care of itself. They’ll want to do what’s best for Apple, not what’s best for them, what’s best for Steve, or anybody else.
We’ve had one of these before, when the dot-com bubble burst. What I told our company was that we were just going to invest our way through the downturn, that we weren’t going to lay off people, that we’d taken a tremendous amount of effort to get them into Apple in the first place – the last thing we were going to do is lay them off.

5. Managing

So when a good idea comes, you know, part of my job is to move it around, just see what different people think, get people talking about it, argue with people about it, get ideas moving among that group of 100 people, get different people together to explore different aspects of it quietly, and, you know – just explore things.
My job is to not be easy on people. My job is to make them better.

6. Revenues

A lot of companies have chosen to downsize, and maybe that was the right thing for them. We chose a different path. Our belief was that if we kept putting great products in front of customers, they would continue to open their wallets.

7. Discipline

People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully.

8. Follow Your Heart

Almost everything – all external expectations, all pride, all fear of embarrassment or failure – these things just fall away in the face of death, leaving only what is truly important.

Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose.

You are already naked. There is no reason not to follow your heart.

Summary

Thank you Steve.

May you rest in peace.

Why We Say “Performance Is A Feature”

John Wang — Tue, 04 Oct 2011 02:26:27 +0000

Singing the Praises of the Unsung Hero

The phrase “Performance is a Feature” has been growing in popularity. Lately, it’s importance for me is underlined by the fact the most common response I get from showing off my early-stage mailing list archive, Grokbase, is that “it’s fast.” I first heard this idea similarly phrased in Google’s Think Quartlerly and then in Jeff Atwood’s Coding Horror which was covered on Hacker News. Matt Brittin, managing director of Google for UK and IE ops said it like this:

At Google, we often think that speed is the forgotten ‘killer application’ – the ingredient that can differentiate winners from the rest. We know that the faster we deliver results, the more useful people find our service.

I’ve worked in some high-performance environments (enterprise search for billion+ document archives) and written benchmark reports so I know performance is important; however, this got me thinking, sure “we” (those in the know) know that performance is important, but what about the meta-question, why is it important to say it is important?

Motivation

For me, the strongest reason for talking about Performance as a Feature is that it can help motivate and focus development. This is important because performance can be vitally important and yet be difficult to achieve and result in little to show using metrics such as lines of code and visible features. Indeed, a product screen shot can look exactly the same after a 100x speed up.

The reason performance can be hard to achieve is three fold.

Performance is (often) Mentally Challenging
Often times, performance is an architectural problem-solving exercise and not the mere act of taking a requirement and executing on it. Additionally, often times you need to taken into account many domain-specific properties in order to fully optimize the system. So you may need to understand the problem space in detail, come up with a solution, and then communicate / convince others that the design should be accepted even before you get started coding.
Performance is (often) not Greenfield
Once you come up with a good solution, often times you cannot just implement it as you are improving an existing process. The new solution needs to integrate into the existing solution and either be a drop-in replacement or change the code around it. Like repainting a car after it gets dinged, the affected code can be much larger than the immediate problem. Further, if data conversion is needed, once the new code has been implemented, the conversion process may take longer than development or deployment.
Performance is (often) Hard to See
Finally, it can be hard to tell your peers what you accomplished if what you accomplished was a performance improvement since there is nothing to see. To complicate matters, you are not necessarily working on a new product or even a new feature. Indeed, you may be just speeding up someone else’s creative work. That the speed-up itself is what makes the work useful at all may be hard to convey. Here, metrics tell the best story to other developers but to people using the app, it may be simply fast.

Summary

As developers, we like to see the fruits of our labor and this often means working on new features that are visible and we can talk about. However, when talking to users, often times the first thing they notice and talk about may be your app’s performance. So when developing an app, it is important to remember, that:

Performance is not just any feature, it may be a killer feature.

Have you had to implement performance features? What were the challenges you faced?

So You Want To Be A Coder

John Wang — Sun, 02 Oct 2011 05:24:14 +0000

Self-Directed Job Enlargement

This article is dedicated to my friends and colleagues working in non-coding technical roles that wish to make a transition or at least increase the amount of coding in their work.

The Dilemma

In software development teams above a certain size, there are often software engineers and systems engineers (including testing, support, sales, and technical marketing engineers). On some of the teams I have been a part of, a number of systems engineers have wanted to become software engineers / coders but face a dilemma. The issue is that if they have not been hired on as a developer, are valued in their current systems role and have not proven themselves able coders, how can they start coding? The situation is often something like this:

A systems engineer wants programming assignments but is not getting any
The development manager realizes the engineer wants to move into coding but does not see enough ability or demonstrable interest to assign coding tasks in lieu of tasks that need to be performed from a business perspective
Without coding tasks, the systems engineer feels he cannot demonstrate ability and thus become a coder

So it seems we have a catch-22 where the systems engineer wants, but cannot find a way, to become a software developer which can lead to decreased job satisfaction.

So what can a systems engineer do? Beyond coding on the side and taking classes which may not impact work at the company, how can one get recognized for coding by your coding peers at the company?

The Do It Yourself (DIY) Approach

While I have never desired to become a full-time coder when I wasn’t hired on as one, I have had my code recognized at some companies and the ways I’ve done it may give you a leg up.

First of all, I don’t view coding as special discipline that only people specifically assigned to code do the coding, after all, most technology companies are not unionized. My view is that code is a tool, much like a spreadsheet is a tool. When you have a task that can use a spreadsheet, use a spreadsheet and when you have a task that can use some code, write some code.

One of the best ways to get started is to write some code to make your work more efficient. This way the code can be easily justified as one way to implement a work task and you’ll get used to coding as a natural part of how you work. If you are in an “Ops” role, you can also view this as a natural extension to “DevOps“. Once you can help yourself, you can move on to helping others and even entire teams. As people see your tools become useful, you will get more recognition as a coder and may formally move into a coding role once you are recognized as such.

As a firm believer in the Do As I Do and Not As I Say approach, here are some situations where I have written programs on the job where it wasn’t my formal responsibility and received recognition:

Be Pro-active
The first part of coding for yourself is to take initiative and not wait until someone gives you a coding task, since you may be waiting a very long time. If you see something that might be interesting, roll up your sleeves and see what happens.

A while back, I would work with our customers on acceptance testing of our product. In this situation, our expertise and value was primarily in the C++ back-end; however our front-end code was available for inspection as it was coded a in dynamic, interpreted language. I thought the code could be a bit tighter here and since I was the one representing my firm on the front line, on my own time, I refactored our front end code. When I showed it to our development manager, it was recognized as good work, and I had a software developer assigned to integrate my code.
Help yourself
The great thing about coding is it can automate and improve consistency for repetitive tasks, making time-consuming tasks manageable and otherwise impossible tasks time-consuming. If you are faced with a task like this, write a program to help yourself out. Creating tools for yourself can have positive spill-over effects as well.

In one situation, we had not yet build an automated installer and I needed to run multiple installations / upgrades of our software on 6 servers. To streamline the then-manual process, I wrote an automated, “one click,” installation / configuration / upgrade script. Not only did others notice I had a one click installation for an otherwise-lengthy task via my installs and upgrades, it gave me some insight that I was able to contribute to our dev team when we did create a formal installer.
Help a Colleague
Once you can help yourself, you can move on to helping others. This requires additional skills as you’ll need to do requirements gathering and push multiple releases if your tool / program is successful. However, the rewards are much greater because now your skills are making an impact on the people around you.

While acting as a product manager ahead of a release, we had to update the localization of our product for a double-byte language where we had significant customers. We were facing significant schedule risk with our process so I worked with our translation and build engineers to create a tool to speed up the translation. We finished on time and the tool was useful enough that I updated it for several releases. Additionally, the colored, graphical status pages made the tool more usable by the translator and more noticeable by others from afar.
Help an entire team and get the buzz out
Finally, when you are ready, you can build an app that is used by many people internally. If successful, your app will go viral inside your organization and people will start creating a buzz about your code for you. The additional challenge from working closely with a few colleagues is that people you don’t normally work with will suddenly start asking you for your app and even have new product requirements you will need to manage.

In this case, the app I built is most likely still in use to derive competitive advantage so I’ll leave out the details. Our team was was working with a Fortune-50 account and I ended up building an app that was used by our team and presented to the client. We successfully closed the account due to many factors; however, it was recognized that this was a significant contribution. Eventually the app became important enough that it was actively requested by our staff (and escalated to higher management when I was otherwise too busy), mentioned by the CEO in all-hands staff meetings and attracted the attention of the CTO who took a personal interest in the app.

In all of these situations, coding was not my responsibility but I took it upon myself to make my life and that of my colleages easier by coding. In the process, my code was recognized by peers including software engineers, systems engineers, salesmen and my CTO. It was also easy to justify spending my time on these coding projects because they improved the work we already needed to do.

While it was not my desire to become a coder in any of these situations, my projects received enough recognition that I believe it would be much easier for a development manager to justify a formal switch or assignments for someone if desired.

Summary

If you are in a situation where you want to code and need to get some recognition for coding before getting a formal role or task, find a project to make your life easier and more effective. If these projects impact the bottom line and if you can get your colleagues using and creating buzz about your apps, you’ll have gone a long way to showing your coding chops.

Have you made the transition from a non-coding role to a coding one? If so, how did you do it? What worked and what didn’t work?

Getting Your Hands Around Email – Introduction

John Wang — Sat, 11 Dec 2010 09:39:53 +0000

A little while back, Craig Ball wrote an article, “E-Mail Isn’t as Ethereal as You Might Think” for Law Technology News which described some high level basics of the MIME Internet mail format standard. Much more technical than the typical LTN article, it highlighted the need for more articles and discussion on the ESI itself. In that vein, here is the first of several articles discussing and examining different email formats. Keep in mind that processing email for E-Discovery may be best performed by legally sound, email management products that have been verified by leading major, independent, third-party litigation consultants.

This isn’t just geek stuff. It’s lawyer stuff, too.
- Craig Ball

Major Email Types Encountered in E-Discovery

Here is a short introduction to the major types of email encountered in E-Discovery.

Internet (MIME/mbox): Virtually all, if not all, mail servers today can handle MIME format email. Open source mail servers often use MIME as their default email format for sending email within the environment and out to users of other mail servers while servers like Exchange and Domino send / receive MIME when communicating outside their deployment. MIME is an open standard defined by the Internet Engineering Task Force (IETF) in several Request for Comments (RFCs). The email format itself is described in RFC-5322. Mbox files are container files for MIME format messages. The basic format is a text file comprising a concatenated list of MIME messages with a special “From line” to delineate the start of each message.
Microsoft (MSG/PST,MIME/EML): Microsoft Outlook’s native email format is MSG, a file format described in MS-OXMSG. End-users often deal with Personal Storage Table (PST) files more often than MSG files; however, many E-Discovery practitioners are familiar with MSG files which often get included with native productions. End-users can generate MSG files by dragging email from Outlook and dropping it on to Desktop or other file system area. PST files are container files for MSG format files. While Microsoft Outlook does not support MIME email, you can read it using Microsoft Windows Live Mail (WLM) or Outlook Express. Simply ensure the MIME mail has the .EML file extension and open it in WLM or Outlook Express.
Lotus (Notes CD/NSF,DXL): Before MIME was established, Lotus created their own proprietary rich data format, called Notes Compound Document (aka Notes CD, Notes Rich Text). NSF files are container files for Notes CD format messages. In Lotus 6 and later, Lotus mail can also be exported as DXL objects.

Email Types in the EDRM Enron Email Data Set 2.0

To get a full appreciation for the different email formats, it’s useful to take a look at some email in the different formats. The EDRM Enron Email Data Set 2.0 supports multiple formats which can be explored. The email was produced by ZL Unified Archive® which can archive / collect / manage email in the various native formats and convert between the various formats as well.

EDRM XML: This is the open standard E-Discovery load file standard as defined by the EDRM XML working group. The EDRM XML files in this data set include ESI metadata along with native email in MIME format (with attachments) and extracted native attachments as well as text extracts.
MIME: While the MIME files are included in the EDRM XML distribution, it is possible to access the MIME without reading the EDRM XML. This has been useful for some research organizations.
PST: All of the email is also produced as PST files for the custodians. These files can be read directly in Microsoft Outlook or processed by virtually all archives and E-Discovery tools.

Email Types in the EDRM Internationalization Data Set

The EDRM Internationalization Data Set provides email in an additional format:

mbox: Mbox files are available in the following languages: Email in the following languages is included: Arabic, Catalan, Chinese, Danish, Dutch, English, Finnish, French, German, Greek, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Romanian, Russian, Spanish, Swedish, Tamil, Turkish.

Closing

I anticipate writing a few more articles on this topic exploring each of the different types of email. It is my hope that layers and other E-Discovery specialists will be able to “grok” email a bit more through these posts.

If you are interested in learning more about these email formats, how to manage them in your enterprise, and how to migrate between them, consider contacting ZL Technologies. ZL Unified Archive® can not only manage email on Exchange, Domino, and Internet mail servers, but it can also migrate email between the different formats.

Image courtesy of: UK Pay Day Loans.

Bill Clinton and Stevie Wonder on Keeping the Passion to Innovate Alive

John Wang — Wed, 08 Dec 2010 16:54:52 +0000

One of the treats of attending large technology conferences is the opportunity to hear inspirational speakers. At last year’s Oracle OpenWorld, I was fortunate to hear James Carville and Mary Matalin so I was excited to learn that Bill Clinton and Stevie Wonder were to speak at this year’s DreamForce conference. As excited as I was, their messages exceeded my expectations and did not disappoint. Being in the 3rd row did not hurt either. While there will be many reporting on their talks, I will report on their messages looking through the lens of business, technology, and innovation, an approach I picked up from Chris Riley at the AIIM Golden Gate Chapter‘s award-winning Tenuta Vineyards wine blending event where we discussed blending ECM technologies the way we blended wines earlier in the day.

Keeping the Passion Alive

A question posed to both speakers was that given their long, illustrious careers, how do they keep the passion alive to drive innovation year after year, decade after decade. This is an important topic for technology firms that try to keep the passion of a start-up in their firms as they grow into larger enterprises. Stevie answered, tongue-in-cheek, “You have to pay the bills.” to some amount of laughter. More seriously, both speakers said it was important to live with passion and be true to who you are. Stevie said, regarding his work, “Music is like life, You have to live with passion for live” and closed with “just be you … you are meant to be you.” Bill remarked on several things that has kept his passion alive:

working on a large idea or concept,
doing something that you both like and are good at, and
having fun at what you are doing (variation of above).

It is no small accident that the most successful people are driven by passion for their work and not simply achieving a certain result. They just love what they are doing. Of course, in today’s global economy and the current recession, it is becoming more important than ever to find that kernel of who you are to differentiate yourself and possibly just pay the bills given the increased competition for jobs. Perhaps Stevie’s first answer is not so far off the mark.

Doing Great Work

The second big takeaway, was the importance of doing a great job no matter what you are doing. Bill mentioned that his foundation is working to help developing countries grow sustainably and to help build systems where people can get good results from good effort and you should strive for good work, no matter your economic condition. He said that “Being poor is not the same as being sloppy,” and also remarked that,

When you build things that work, good things happen.
- Bill Clinton

For start-ups, it means that even when you may have few resources, you still need to put out a top notch product. Sometimes, you may need to cut back on features to ensure all the features you ship are top notch.

Great Work is Not Enough, You Have to Advertise!

Bill also lamented his party’s performance in this past mid-term elections. Specifically he mentioned that although his party had raised substantial funds, they did not use those funds to tell the story of their successes resulting in losing many more seats than were forecast and losing control of Congress. He did not blame their adversaries but looked within said that it was his party’s fault that the electorate simply were not informed of the great work that had been accomplished.

From a technology company perspective, this highlights the need for and importance of marketing to communicate the great product being built by the development teams. From a product management perspective, these are linked to drive a company and product’s success. The Pragmatic Marketing educational organization lists both product management and product marketing as product management disciplines. Reflecting on the importance of product management, they now list executive leadership as a product management discipline, a topic for another post.

Being Forever Young and The Business of Tomorrow

Looking at the US and its chances for remaining a dominant country with the rapid rise of countries such as China and India, Clinton mentioned that he liked America’s odds but that to be successful, the country needed to be forever young, that “we have to be a tomorrow country” and that the US needs to become a “Laboratory of Democracy” as a country, something once said of state governments.

We have to get back into the tomorrow business
- Bill Clinton

Along with moving back into the tomorrow business, he said that “everyone’s job is to work themselves out of a job,” a key message that tomorrow’s jobs will be vastly different than today’s jobs. He highlighted in the move to IT jobs over the course of his Presidency when such jobs comprised just 8% of all jobs but 30% of job growth and 35% of income growth.

One way to apply this to the technology field is to look at the rate of relentless innovation at 34 year old Apple Computer which, according to the article, is driven by three key factors:

It invests heavily in R&D
is unafraid to cannibalize or kill its own products and
is able to extend its core technology across a host of different products to create a dominant ecosystem of consumer gadgets

These are similar to Bill’s approach on working one out of a job and being in the business of tomorrow.

Recap

The speakers were thought provoking and inspirational and I am glad to have been there. My article is certainly a selective approach to discussing these talks, however, I think this format presents some key messages from business, technology, and innovation perspectives. My thanks to the folks at SalesForce.com for bringing us such great speakers.

Image courtesy of: LIFE.