Spring Cleaning Time for SharePoint Sites

We have been deeply involved in a cleanup of our SharePoint portal plan for the last several weeks and I thought I would share with you some of our observations because I know most of you with both Internet and intranet sites that are more than a few years old probably share the same issues we do. Our Internet and intranet portals have been around for 9 and 8 years respectively and over time, a lot of ‘junk’ has accumulated. The example I like to use is that it is similar to the way your email slowly fills up with ‘junk’ over time. For example, if you received 20 emails a day, but only have time to address 19 of them, you may go home feeling pretty good about the amount of work you got done. However, that 1 email you did not get to if you consider have only 1 email that you do not get to each day over the course of a typical work year can amount to over 250 emails. Of course, if you get more than 20 emails and the number that you do not get to is greater than 1, that total can expand much more rapidly.

So how do you get a handle on the problem? You could simply delete anything older than 2 weeks old, but that might delete important email messages that you really need to see. On the other hand, the email that you received yesterday and did not get to may be a total waste of time. Clearly the age of the email is not the only deciding factor. Maybe you choose to delete all emails from people outside of your department. Unfortunately, some of those emails from other departments may be more important than the email that circulated around asking people where they wanted to go to lunch on Friday. You could delete anything that comes from outside of the company. That would certainly help keep you focused on your work, but you would also miss notifications of appropriate training or white papers relevant to your job.

Thus you can see that cleaning out your inbox can be more complex than any simple rule or a set of rules (although they may help). It is also important to perform that cleanup on a more frequent basis than once every year. One a month may not be too often and even once a week perhaps on Friday afternoon as you are winding down for the week might be a good choice.

In a similar way, if you have both an Internet and intranet portal (hey, even if you only have a collaboration site), periodic cleanup is still something you need to consider doing. If nothing else, cleaning out old obsolete content will make search run faster and return more relevant results. So here are a few tips that may help you perform your own portal cleanup.

10 Steps for site owners to consider when performing their next portal cleanup:

1. Remove obsolete or unnecessary sites – Sites where all content pages/documents have not been updated for 2 or more years are candidates.
   
2. Examine all pages for duplicate or obsolete content and update or remove – This could result in removing the page itself if all content on the page is obsolete and removed.
   
3. Remove obsolete/duplicate documents/files – Multiple instances of files all get indexed and results in bloating the search results with many invalid entries that do not point to the most recent data. Delete obsolete/duplicate files.  Burn copies onto a DVD if you want to keep them.  Adding them to your intranet site or collaboration site is not a valid solution.
   
4. Remove content that appears on other sites for your organization that you do not own – Copying/duplicating content that appears on other sites within your organization bloats search results and diminishes the relevance rating of the correct document if multiple occurrences exist.  Any content not ‘owned’ by the department should be removed and replaced with a link to the content on the ‘true’ owner’s site.
   
5. Remove content found on sites outside your organization – Not only is this a potential copyright issue, but updates made to the content on the ‘true’ source will not be reflected in the copied content resulting in misinformation.  Just link to external content.
   
6. Clean out your calendar/announcements – If your site has a calendar or an announcements list, clean out old events that are no longer relevant.  This will improve the performance of the calendar and/or announcement list.
   
7. Consolidate sites – Sometimes subsites were create when all that was needed was another page on the site that owns the subsite (parent site).  Unless the subsite requires a different set of permissions (owners, content managers, approvers, etc.) you may be able to simplify your site structure by moving content/pages/documents up a level.  This will also improve navigation and reduce the number of clicks to find the content you need.
   
8. Remove content that really does not need to be public – For any content item (subsite/page/document) ask yourself if anyone in the public really needs to see this content on a public web site or whether it just clutters the public facing sites with content that no one really looks at.  Perhaps all you need is a ‘Contact Us’ link for anyone in the public to request additional information if necessary.  Some current public content probably should only be internal intranet content.  If so, move it there if it does not already exist and delete the public version.
   
9. Do not duplicate content between the Internet and intranet – If the content needs to be seen by both the public and organization’s employees, place the content on the Internet and only add links to that content from the intranet. Don’t place the content on both and definitely don’t place the content only on the intranet.
   
10. If content is not owned, remove it – If you have current Internet content that is not officially owned consider removing it.  Content that is not owned probably is not updated.  If the content is necessary, an owner for the content must be identified.
    

Well, that’s it for this time. C’ya.

Only at Participating Locations

Ever wonder about this phrase often found at the bottom of advertisements, coupons and promotions? Did you ever think about what it really means or how it can be used? Or how about the phrase, ‘Participation may vary by location.’ Just what does that mean to you and me? There was a time I assumed that any business chain with multiple locations around the country owned by a single parent company had no choice but to follow all nationally advertised promotions. I also assumed that a business that was franchised had to follow any promotions of the franchiser, or parent company so that all customers would have a common experience no matter where they went. If you live in a small town, you may not notice the difference between how one business location operates compared to another because you may not have that many locations. Furthermore, you may think that if the parent company is spending large sums of money to develop and advertise promotions to drive customers to your door that you would want to take advantage of that business hoping to convert at least a percentage of them into repeat customers.

But apparently, that is not the case. In fact, I’ve notice quite a few businesses who advertise on television and direct email but when you go to their local stores, they do not honor the promotions. Perhaps the local stores feel that people might feel guilty about just getting up and walking out when a local store manager says, ‘Sorry, we are not participating.’ But are they really sorry? If they were so concerned about our feelings they would not trick us into their store only to say that they are not participating. Maybe it is time that we walked out of their store.

Truth be told, I live in tourist town, Orlando, FL. I first notice this effect several years ago. One time when we questioned the policy we were even told that because they are a tourist location, they do not honor the advertised discounts. I suppose they figure that the tourists have no choice. Most cannot make meals in their rooms. Most do not know the area well enough to risk finding another location that may be participating. And most don’t want to take the time away from their vacation just to save a few dollars. So you might think of it as a convenience fee.

Besides food chains, this effect becomes even more obvious with the price of gasoline which seems to almost consistently go up the closer you get to the major tourist attractions or the airport and its surrounding car rental agencies. Yes there are some exceptions and the locals know where they are just like the locals know the best places to eat that are not part of nationwide chains.

But let’s get back to the main issue. I know several of the fast food chains with locations near the tourist attractions that will not honor the promotions of the nationwide chain. Several times I’ve been told point blank that the locations near the attractions can charge more because they are there just for the tourists, not the locals. Were they telling me to go away? I guess I moved to the wrong side of town.

A good example occurred recently at a nationwide chain that was offering a meal deal of an entrée with an appetizer. (I will not go further into the details here.) The promotion was sent directly to my email address because I subscribed a long time ago to their ‘club’. Granted, the promotion added the words, ‘Limited time only.’ The thing is, I received that email just last Thursday. The next day, Friday, we decided to go there for supper to take advantage of the deal. As we sat down and were given menus, I noticed that there was no reference to the ‘special’. So I asked about it thinking that maybe they only gave the promotion menu to those that asked. Instead I was told by the waiter that he had heard about that promotion, but thought that it had ended but would go ask the manager. A few minutes later he came back and told us that his manager said that the promotion indeed had ended. Fortunately I could still bring that email from the previous day up on my phone. Aren’t smartphones great? So I showed him the email and the date on the email. He went back to his manager to ask again. This time when he came back, he said that his manager said, ‘We are not participating in that promotion.’ So we got up from our seats. He asked us if we were leaving. We replied that we were not participating in bait and switch tactics and left.

Really makes you wonder though whether the promotion was only for one day (which it did not say in the email) or whether they just did not want to offer it to their other customers, the majority of whom where tourists.

In any case, I say that it is bad business to advertise any promotion and then decide not to honor it just because your location is near tourists, a more affluent part of town, or any other arbitrary reason. It may be a long while until I go back to that location. Maybe they don’t care. Maybe they get enough business from the tourists. For me, I’d rather find another location where the location manager operates fairly and honors the promotions of the parent organization he is a part of. If you don’t like the promotions, become a store manager of a store that does not have promotions or start your own store if you want to do whatever you want. The bottom line for me is that unless there is a legal reason not to honor a business promotion created for the chain that is designed to send more customers to your door, you should honor the promotion and give those customers the best experience possible so that they might decide to come back again and again.

C’ya next time.

When Counting the Beans Is More Important Than What the Beans Mean

It seems like today everyone wants to measure their business processes.  If it cannot be measured, it is not important.  In general that is not a bad practice because all businesses whether they are multinational corporations or Mom and Pop corner stores needs to know at least a few basics about their business to determine if they are doing well or are about to go bankrupt.  But the fact is that many newly minted MBAs focus so much on counting everything that can be counted, that they lose sight on the long term goals of the business or trends of the industry.  Often this is a lack of keeping an eye on large scale business trends and being aware of how customer tastes and purchases are changing.  Then when business drops off, they scramble to find other reasons for why the business is failing.  This is what I mean by being too concerned with counting the beans rather than what the beans mean.

Not too long ago, one of the top businesses many people tried to get into was renting movies through local brick and mortar video stores.  Demand was growing as people moved from VHS tape machines to watch their movies to DVD discs.  The number of available movies grew.  Total rentals were up.  Profits were up.  Some movies made more from DVD sales than box office sales.  What could possibly go wrong?  Then it seemed like the bottom dropped out of the market as first local Mom and Pop stores closed and then the chains began to close.  Did people stop watching movies at home.  No.  If anything people watch more movies than ever.  The difference is in the way people get their movies.  Most TV cable companies and even Satellite companies now provide movies on demand.  You can rent movies from many local libraries for free.  Some of the big online retailers like Amazon provide movies on demand.  Why get into your car to drive down to the local video store, look for a video (only to find the last copy has already been rented), bring home an alternate movie that you really did not want to see but felt you had to get to justify your drive to the store, watch the movie, and then rush back to the store just before it closes to return the movie on the last day it is due.  Stores missed the convenience factor that has taken over the market.  While counting their beans, they missed the fact that their beans were changing.

A similar argument can be made for bookstores.  At one time, the only way to reasonably get the latest best seller or a technical reference book was to go down to your local book store to buy the paper version of it.  It was not all that long ago that every mall had at least one bookstore.  Now many malls do not even have one dedicated book store.  Many small bookstores have closed shop because they could not compete with the large chains.  Then some of the large chains even started to fail.   Some tried to counter the trend by adding lounge areas where you could grab a book and read a bit of it before deciding to buy it.  Some added cafes.  Some included live music on select nights.  But sales still continued to fall.  Again the driving factor was ordering books on line could often be accomplished from the comfort of your living room chair and you could get the book delivered.  Some libraries also added home delivery of ordered books.  But I believe one of the big game changers here was the introduction of digital book readers.  Sure the early ones were bulky, heavy and limited to monochrome text and little to no graphics.  But these quickly evolved until today the quality of digital books displayed on light weight tablets and even phones rivals the quality of printed books.  Sure many people still like the feel of a real book in their hands, but as the ebook versions of popular best sellers came down in price and more technical books became available in digital form, the demand for digital books has grown.  I confess that in the past two years, I have not bought a single paper book, but I have purchased at least two dozen digital books, often at 50% of the cost of the paper versions, and I could download them to my device immediately.  Many classic books are available free and at least my public library has been ‘loaning’ digital copies of books on-line for several years now so I do not even have to go into the library anymore.

Both of these examples clearly show the demand for the end product growing yet the distribution method for these products have changed dramatically.  Some libraries understand what the change has meant and they have adapted to it.  Clearly, they understand what the beans mean.  Others who have been too busy counting the number of patrons or the average number of books borrowed or the total size of their collection have lost track of what the beans (reading books) really means to the public.

How is your business adapting.  Are they busy counting the beans in the current month, quarter, or year?  Or do they spend time trying to understand and predict where their business I going in the next year, 5 years, or even 10 years.  Are they preparing for that future or do they assume that tomorrow will be the same as today and success is simply counting their current sales, profit margin, customers, etc.  I’m sure you could look at other industries and make similar analogies.  What is happening to newspapers and magazines, professional photography, or even education?  Is the method of delivery of your product or service changing?  Do you really think you will be doing the same job in 5, 10, or 15 years?  Knowing which beans matter and which can be ignored might be a more valuable skill than merely counting those beans.  Many people have made careers counting beans or having others count their beans for them.  In the meantime, other organizations more interested in what the beans meant have surpassed them and will (if not already) threaten their existence.  Some express surprise when companies or industries fail.  Others express surprise when new companies they never heard of before succeed.  But now you know the real reason why companies succeed and fail.  After all, any simple computer (or average math student) can count your beans.  But can a computer program tell you which beans matter the most for your future or your company’s future?

Remember to come out to SQL Saturday Tampa this Saturday.  C’ya next time.

Getting Back Into the Groove

I first want to apologize to my regular readers for being ‘absent’ for much of the last two months or so.  The last six months have been some of the hardest, yet some of the happiest, (for very different reasons) in my life.  The hard part as those who know the situation took up most of the first half of the year, but the happiest part is seeing my daughter move from a residency program at a local VA hospital to a full staff member as of July.  Her success after going through all the same hard times as me gives me hope.  So the last month we have been spending time looking for a new apartment for her, looking for furniture, packing what she wanted to move over for now, carrying stuff over in cars every night and unpacking it into her new place and even the ‘joy’ of building IKEA furniture.

She didn’t move far.  Just across town.  Far enough to have her own life, yet close enough for visits.  I don’t know if she truly realizes how proud I am of her accomplishments, but I am.  I don’t know if we were just lucky or if Sue and I (mostly Sue probably) did something right in raising her, but for the life of me I cannot figure out what that may have been.  So don’t ask for advice.

Anyway, I suppose it is time to start over and to start writing again.  After all, the house is empty now except for my cockatiel and before her cheeps start to make sense, I think it is time I actually start typing real words.  So, over the past week I’ve picked up paper and a pen and started to write down several ideas and I’m sure some of them will eventually develop into blog entries.  For today however, I just want to leave you with some random thoughts that will probably never make it to a full blog entry.   (By the way, the bird is sitting on my shoulder right now watching everything I type so I have to be careful what I say.)

I’ve noticed in the newspaper lately that a lot more motorcycle accidents are reporting that the injured riders were not wearing helmets.  Did you ever wonder why the government was so concerned about whether we automobile drivers have a seat belt on while we are driving and will even fine us several hundred dollars if they stop us and we do not have a seat belt on yet motorcycle riders don’t have to wear any head protection?  Seems odd.  Maybe it is a Darwin thing and we just want to thin out the population of those whose heads are too thick to be injured during an accident where they lose control.

I’ve also noticed that every major storm, hurricane, tornado, dry spell, heat spell, etc. has been linked by the media to global warming.  (Some people even think that Sharknado was a real documentary and was caused by global warming.  Some of these same people also ride motorcycles without helmets.) They act as if extreme weather never occurred before they discovered global warming.  While there may be some truth to the connection, their level of conviction that they are right and that everyone who does not agree with them is wrong or perhaps stupid seems to put people off who might otherwise at least consider the possible connection.

Currently the big story is the Royal Baby in England.  I will grant you that for those people living in England or originally from England, that is probably a very important story.  But really, this baby (I don’t think he has been given a name yet or at least I have not heard it) is only something like third in line for the throne.  On the other hand, that is probably a lot closer than you or I will ever be.  Anyway, Cheers! to England.  At least they have some good news in their media for a few days making it worth watching the BBC.

I guess Earth missed getting hit by a asteroid the other day.  It was suppose to be between 200 and 400 feet long.  Let say something around the size of a football field.   Unless it totally broke up or burned up during entry into our atmosphere, I suspect it would have made quite a dent in your car if it fell onto it.  The amazing thing was that it was only discovered a few days before its closest approach.  Yet we are being told by astronomers and the government that they have mapped over 90% of the trans-Earth orbit asteroids and would know well in advance of any potential problems.  No wonder they didn’t notice Clark Kent’s spaceship during that meteor storm.  I guess that few extra percent can really make a difference, especially if the asteroids targets your local corn field.

Finally, I leave you with this thought.  I’ve noticed that average employees who leaves a company to go into consulting becomes an expert overnight in whatever field they are talking about as long as they charge more than $200 per hour and travel at least 500 miles to the client site.  Similarly, I’ve seen consultants get hired by a company for their expertise in some technical area and overnight in the new job become dumber than a door nail (whatever that is) whose opinion is not worth a wooden nickel.  I guess it just goes to show that knowledge in any area is fleeting.

This Saturday, I plan to pick back up with a technical article on Data Quality Services where I left off months ago showing how to create a matching policy to find duplicate records.

Till then, c’ya.

Overpromise and Underdeliver

“Wait a minute,” you say, “you got that backwards.”  Do I?  What was the last major purchase you made as a result of a marketing person or perhaps a salesperson  touting how great the features of that item were and how it would transform your life.  Did it?  If it did, congratulations, but most of the time we feel a sense of disappointment after buying something expensive that does not live up to the hype we were told.  Perhaps it was that sports car that was going to attract all the girls, but instead attracted higher insurance premiums.  Perhaps it was that computer with the new dozen core processor promising to cut our work time to a fraction until we found out that our software only takes advantage of a single core no matter what.  Maybe it was the new cell phone that would allow you to eliminate the need for a home computer because it could do anything and everything a PC could do.  Then you found out that half of the memory was taken by the OS and pre-installed software and the rest of the memory was not only not expandable, but that only certain things could be relocated to the micro-SIM card.

I maintain that everyday people are out to sell us something by over-promising the features, the savings, or the efficiency of their product.  In the meantime, the manufacturing portion of the company appears to under-deliver the final result.  I suspect that it is not always the manufacturing department’s fault.  They may have built a very good product based on the specs and requirements given to them at the time the product was first designed.  However, in order to sell that product and meet their sales quotas, the marketing staff has to stretch the capabilities of the product just a little to make it sound like a better choice than a similar product from that other company down the street.

Did you ever notice how some movie trailers give away too much in the desperate desire to convince you to come to the movie theater so they can sell you a bucket of popcorn that could feed a small town in Africa and a soda that costs as much as a half dozen 2-liter bottles?  Some theaters even allow you to get free refills on that bucket of popcorn loaded with melted salted butter.  Why?  So you are thirsty enough to buy a second round of sodas of course.  Put aside that popcorn bucket with its ton of butter and riddle me this, do movie trailers sometimes give away so much of the most interesting action scenes, even some of the plot twists, that they effectively have overpromised the enjoyment you will get out of seeing the entire movie which then appears to under-deliver the entertainment value?

I just went with my work team on a team-building event to see the latest Star Trek movie.  (Yes it is a valid team builder because everything you need to know about working together as a a team you can learn from Star Trek.)   We made a point of not listening to the reviews before going and we tried to avoid the advertisements and trailers although we were less successful at that.  I’m not going to spoil the plot for you, but let me just say that the trailers in this case did not spoil the twists in the story line.  In fact, in some ways, we were totally misled as to what was going to happen by the trailer.  Whether that was accidental or on purpose, I do not know.  Go see the movie and then let me know here.

However, this movie and indeed, most of the entire Star Trek franchise revolves around the concept of under-promising and over-delivering.  No matter how badly the ship is damaged, Scotty somehow manages to pull it through.  Ok, he did admit back in one of the movies that the way he was able to maintain his title as a miracle worker was by doubling his estimates.  Also, no matter how high the odds are stacked against Captain Kirk, he somehow always manages to save the day.  In fact, the heroes in most movies have that same characteristic, the ability to over-deliver when it really matters.

How does this apply to your IT job?  The last time someone asked you to estimate the cost (time and materials) for a project, did you sit down with a few of the developers to design the system on paper to arrive at a realistic estimate of how long it would take to complete?  Did you take into account that the average developer may not actually be productive for a full 8 hours in every single day they are at work?  Did you factor in vacation time, sick time, holidays, the inevitable time lost due to meetings, and the need for real testing, documentation and training of end users, not just the time needed to develop the initial system?  How about adding a contingency on top of that for the changes to the project’s scope?

Sure it’s not a perfect science.  It is difficult to predict major impacts to a project such as key staff members leaving the company or the ‘special project’ that supersedes all other projects that the president of the company wants done yesterday.   Estimating is really an art that a manager only learns by experience, by knowing what their team is capable of, and by being able to look at the bigger picture, not just the lines of code needed for specific functions.

For that reason, you may want to under promise the features that the new software will support or the amount of time needed to build and deploy the software by carefully examining the needs and the wants.  Trying to do everything wanted may cause the project scope to expand turning it into Mission Impossible which for most companies is equivalent to the over-promise/under-deliver trap I mentioned earlier.  Rather postponing some of the wants until a version 2 might be a smart way to get a system out in front of the user to get a better idea based on their user experience which wants should have a higher priority in the next iteration of the software.  By delivering something that is useable and meets all the essential needs early rather than delaying the development to implement all the wants can buy your team user acceptance and credibility much like Scotty on the Enterprise.  And if you have time, add back some of those wants that you did not promise in the initial system.  In other words, over-deliver.

Finding Out When Enough Is Enough

Last week I asked the question, ‘How many people must you ask in a survey?’  While I talked about the topic in generalities at that time, I also mentioned that it would be interesting to test the hypothesis that a survey only needs to query a small percentage of the population to get meaningful results.

To test that theory, I took some data from a recent survey that was conducted over several months and disguised the question, but kept the results.  The question was a basic Likert Scale type question in which the question itself postulates a specific position and asks the survey take whether they agree or disagree with the statement.  This survey was conducted using the SharePoint Survey list and was set to allow a user to only answer the question once so as to not pad the ballot box so to speak.

The  total possible population of respondents was around 12,000.  Of course the survey owners wanted to get as many respondents as possible which is why they conducted the survey over several months.  However, I have always been of the opinion that for this survey, anything more than about a month of making the survey really did not serious effect the overall results.  By that I mean that the responses after about a month were a true representation of the total population and that there was no need to try to get 100% participation.

However, the question I chose to use for this study contained five possible responses listed below:

• Strongly Agree
• Agree
• Neither Agree nor Disagree
• Disagree
• Strongly Disagree

Since I was collecting the data with SharePoint, I also stored the date on which each survey was taken.  Therefore, I could tell on any given date, how many responses have been entered since the start of the survey.  Knowing the total population, I could very easily determine the percent participation.  By exporting the data from SharePoint to an Excel spreadsheet, an extremely valuable option from a SharePoint survey, I could load the data into a PowerPivot data model and then create a variety of tables and charts based on the data.

The first figure I show below is the final tabulated results  after three and half months of data collection.  You can see that the total count of responses was only 8,203 out of a possible 12,000.  This represents a little more than 67% of the population.  Of the people who responded to the question (Yes, the question was changed to protect the guilty), ‘I believe Pivot Tables help me analyze data at work, 63.7% of them strongly agreed with the statement.  In fact, over 96% agreed or strongly agreed with the statement.  But my question was, did I need to poll 67% of the population to discover that?

Going back to my PowerPivot table, I added a report filter (For those that don’t have PowerPivot, this data set is small enough that a simple Excel Pivot table would also work fine.).

When I opened the filter dropdown as shown in the next figure, I can expand the All node of the value tree to show all the possible values in the table.  Note each date is represented as a separate entry.

In order to select multiple dates as my filter, I need to click the checkbox at the bottom of the list box: Select Multiple Items.  This action places a checkbox next to each date as well as the All node.  By default, all records (dates in my case) are selected.

I first need to unselect the checkbox next to the All node.  Then I can select only the dates that I want to appear in my table.  For example, in the next figure, I select only the first three days of the survey.

When I click OK, my table updates and shows a total count of 214 survey responses on which 76.64% strongly agreed with the statement.  While this is close to the final 63.7% at the end of the survey period, it is still 13% away.  Obviously 3 days of a survey are not enough.

I then chose 10 more days through February 2^nd.

This time with 1103 responses, my results for strongly agree was 65.55% and my total for strongly agree and agree were 96.7%.  Now I am getting really close to my final results and after only 13 days rather than 3 and a half months.

I added another 10 days bringing my survey count up to 4023, nearly half of the three and half month result and my Strongly Agree percent is starting to settle in at 63.81%, only a tenth of a percent off of the final result.

So, just for fun, (statistics is fun isn’t it?) I decided to chart the percentage of Strongly Agree responses as a function of the survey date.  I noticed that by the time I hit a month into the survey, my results had flattened out to around 64% plus or minus less than a half percent.

I then plotted the percent response rate assuming a maximum of 12,000 possible responders and to only about a 15-17% response rate.

So after surveying only about 15% of the population, I could say that the additional survey results over the next two and half months would not significantly affect my results.  Therefore, I could also say that it would be reasonable to assume that even though I only surveyed 67% of the total population, getting responses from the remaining 33% would probably not significantly change my results.

That is the power of surveys.  The trick is determining when the survey results begin to flatten out.  Every survey can be a little different and the number of possible answers to the survey will also affect the result (something we can maybe test in a future blog entry).

If I were plotting this data on a daily basis, I would have been able see when my results began to flatten and be able to ‘declare a winner’ with a great degree of certainty after a month and half or perhaps less.  In fact, with greater experience with similar types of data and by using questions with fewer possible answers, the size of the survey can be greatly reduced while retaining a high level of accuracy in the result.

I hope you found this interesting.  I chose to give the Tuesday blog a bit more of a technical twist this week because I am about to go on a summer writing schedule.  What does that mean?  I may drop back to one blog entry a week for most weeks.  There is just so many other things to do in the summer that are more fun than writing a blog, like cutting the grass and pulling weeds from the garden or even trimming overgrown bushes.  Anyway, I’ll try to keep a few non technical blogs in the mix each month to lighten up the reading from the dry technical topics.  When fall comes, I will switch back to two entries a week.

C’ya later.

How Many People Must You Ask in a Survey?

Have you ever conducted a survey to help make a decision?  Perhaps you helped someone else build a survey to help them make a decision.  I’m sure you have taken surveys.  I seem to get several every week in my email from various companies and organizations.  Then there are the surveys on the back side of your receipts from restaurants and stores.  You might even get surveys in the mail.  I remember when they use to include a nickel in the envelope with the survey to make you feel guilty about keeping the nickel and not filling in the survey.  I guess a nickel doesn’t buy a lot of guilt today.

But how many of the surveys do you think they get back?  Ninety percent?  Seventy five percent? How many people need to respond to a survey in order to make valid predictions?

I guess the answer depends on who you send the survey to and what questions you ask.  If you are a auto manufacturer and you want to evaluate your customer satisfaction with a new model, you do not want to send the survey to all automobile owners.  On the other hand, if you want to find out what features would entice owners of automobiles from other manufacturers to switch and buy one of your vehicles, you may want to exclude owners of your cars.

You see the dilemma?  The type of question should be closely tied to the audience to whom you send the survey.  If you cannot narrow down your audience, perhaps the questions in your survey are too broad and you should consider making two or more surveys to target specific audience groups with question that would be important to them.

How many questions should you put on a survey?  The more questions you include, the less likely someone will take the time to answer them.  For myself, anything more than a half dozen questions and I’m bored and ready to stop taking the survey.  One way you can counterbalance this tendency is to offer greater rewards for completing the survey.  Restaurants often offer free appetizers or deserts or even menu items for completing their surveys.  Stores may offer a certain percent off your next purchase.  Internet surveys have offered everything from cash/gift cards to thumb drives and even iPads.  Would you fill in a 100-question survey for the chance to win a 4 GB thumb drive?  What if they were offer an iPad for 5 lucky survey takers?  You might want to guess at how many people are willing to take that survey.  If you think only 500 people will take the survey, you might be more willing to spend the next half hour completing the survey than if fifty thousand people were to take the survey.  On the other hand, a survey that offers nothing in exchange for my time will probably wind up getting filled in my circular temporary storage bin otherwise known as a trash can.

So you have identified your survey questions and you have a targeted list of people you will be asking to take the survey.  You even have a reward program set up to encourage people to trade their time for a chance at a ‘gift’.  What percent of your target audience do you need to get a response from in order to have a reasonable chance that the survey will represent the total population?  Take the last presidential election as an example.  Did any of the survey takers ask you whom you might be voting for?  Probably not.  In all of the years that I’ve been voting, I’ve never once been asked by a survey taker who I was going to vote for or who I did vote for.  So whom are they asking?  Would it surprise you to know that they can get a pretty good idea of the way the election will turn out by only ask a few thousand people?  Makes me wonder if we just cannot randomly select a few thousand people from around the country to cast their ballots instead of trying to get all of us to drive to the polls and then wait in lines for hours to cast our votes.  If the odds are that we would get the same results, imagine the time we would save.  After all, isn’t that what manufacturers do when they conduct product surveys to determine what goods to make and sell to us?

I was recently involved in a survey with a very well defined maximum population.  The survey was open for several weeks, but I can evaluate the cumulative data as of any day beginning with only a percent or two up to the final 60+% results.  The survey owners were still trying to get people eligible to take the survey to complete the survey before the deadline even though they already had survey results from over 60% of the eligible survey takers.  Was this overkill?  Would a few more percent make a difference in the outcome of the decision?

Over the next couple of days (or weeks if I get busy doing something else), I’m going to pull the data into a PowerPivot model and evaluate the results to some of the questions over time as more and more people take the survey.  I’m curious to see if getting more people to take the survey really made a difference.  I’ll report back to you what I discover in a future Tuesday blog.

Until then, think about it and try to reason out what the results will look like.  C’ya!

 

Wisdom, Not Brilliance, Will Make a Difference

Ok, it has been a tough week for me, but I need to get back into things.  I recently was catching up on listening to some of my older webcasts.  One in particular from back in early 2009 caught my ear and I thought I would share some of the insights I got with you.  The specific webcast was part of the TED Talks series and was presented by Barry Schwartz.  It was a very passionate presentation because Barry really believes in the  importance of his topic.  However, I know people who would call him angry and mad, unable to control his emotions, and danger to have around ‘real’ people.  Unfortunately, these people don’t understand the difference between passion and anger.  That is too bad, because passion is often what drives progress.  Well, let’s see what Barry says.

Barry began by talking about the typical job description.  Most jobs simply list the things that the individual should or sometimes should not do.  They rarely if ever go into interpreting the way the person in that job should interact with others.  Oh sure there are simple comments like the employee should work well with others, but what does ‘work well with others’ really mean?  Does it mean that the employee should simply follow the rules given to them by their boss and walk lock-step like an android with never any additional thought about why the rules are there or whether there should ever be a reason to disobey a rule.

Barry tells several stories such as the janitor who stopped mopping a floor at a hospital even though he was told to get it done now because he saw a patient trying to walk up and down the hall with a walker after an operation.  Another hospital employee refused to vacuum the carpet in the waiting room because there were some people in the waiting room who had been up all night with a sick relative and were trying to catch a little nap before going back into their family member’s room.  While these were hospital related examples, I remembered them because I related to how nurses would come into a darkened room in the middle of night and turn on the bright lights just to take a patient’s vitals.  Can’t they have a lower wattage night-light rather than waking up the patient every 2 hours? I thought sleep was suppose to be healing?  Can’t the vitals be obtained remotely?  Or how about the buzzers and alarms on the electronic equipment at night?  Do they really have to be that loud?  Isn’t it possible in this day and age to have them automatically signal the nurse’s station or send a text message directly to the nurse’s cell phone so they can get the message no matter where they are?

Where is the kindness, caring, and empathy in today’s world?  Where is the moral will to do what is right rather than simply what some procedure says to do or what will cost the least amount of money?  Having to make many phone calls to various people lately I can tell you that I am sick and tired of answering machines that pick for people that are either not at their desk, on another call, or simply do not want to answer that say, “Your call is important to us.  Please leave a message and we will get back to you as soon as possible.”  Four hours later you might get a call back.  Sometimes it is the next day.  In either case, I have to ask how important was my call to them really?

A wise person knows when to ignore the rules and when to improvise for the greater good of the customer, patient, or client.  They treat rules as guidelines, but not absolutes, not as limitations.  They depend on their experience to choose the better path.

Barry then goes on to tell the story about a father and son at a baseball game.  I actually remember seeing this in the news.  The son was thirsty and wanted a lemonade.  The father went to the concession stand and bought a Mike’s Hard Lemonade which was a relatively new product at the time.  He really did not know this product had alcohol in it.  (I see you snickering.) Anyway, he brought it back for his son and an employee of the stadium saw the boy with the lemonade and called the police.  The father was arrested and they tried to move the boy to foster care.  It took two weeks for things to be straightened out and the father reunited with his son.  The point is that a simple conversation could have solved the issue without all of the expense of an ambulance, police, courts, foster homes, judges, and more.  Wisdom would have solved the problem in minutes while procedures cost everyone time and money for what was a simple mistake.

Brilliance is nothing without wisdom.  Even the most brilliant person can look stupid if they don’t apply their brilliance to know when and how to apply rules.  Take away a person’s ability to think and just blindly follow rules and you take away the growth of their moral skills.  Furthermore, too many rules can lead to stagnation.  Providing incentives on top of those rules takes away a person’s understanding of doing what is right unless they are paid for doing it.

Barry also maintains that we all need to build character starting with students in our schools.  We need to teach them how to respect themselves, how to respect their school mates, respect their teachers and staff, and respect learning.  Everything, he maintains, follows from that.

Is Barry right?  Perhaps.  At the very least, it should make us pause to consider.  All I can say is that it is all about what you do and how you do it.  Practical wisdom, not blind obedience to rules, will help you make a positive contribution.  That type of wisdom does not require brilliance, but it does require practice building your moral skill and moral will.  If your organization does not support you building those skills, then even the best employee forced to constantly swim upstream will give up and never really soar with the eagles.

Thanks Barry for a very insightful presentation.

C’ya next time.

Knowing When It Is Time To Start Over

Perhaps you have been in IT for a long time or perhaps you are just getting started.  In either case, one of the hardest things to learn is knowing when to start over.  If you have been in the business for any reasonable amount of time, you probably had the misfortune to take over someone else’s project after they left the organization.  Maybe they left on their own.  Maybe they did not.  Maybe the application was written by a consultant who came in and just took the money and ran as soon as the last line of code was committed. Maybe the code has been all yours from the very beginning but due to a lack of planning, the software tends to meander around with a lot of excess code, obsolete routines, and poorly written code.  Maybe it is not your fault.  Maybe the client could not make up their mind about what they wanted and had you re-writing the same modules over and over with different changes.  Maybe the code was well written in a language that is no longer available or multiple versions behind the current release and even though you have the source code, there is no way to compile this old code using modern language compilers without substantial changes.

All of these scenarios and probably more should make you stop and at least consider whether it is a good time to abandon the old system and start anew.  This is often not an easy decision especially when management uses the argument that they cannot afford to abandon the old system because they stuck so much money into that system, that they would look foolish to just walk away from it.  However, they may look just as foolish if they continue to throw good money after a system that  may not be worth salvaging in the first place.  The point I want you to consider is that any decision should not be based on what has happened in the past, on how much money was spent on a system that is now floundering, or on personalities that may have been involved in that project.  The decision must be based solely on the current value of all future costs of each possible project path into the future.

Suppose you had a program that was costing you $12,000 a year to maintain this year, but the cost of contracted labor to support that program is expected to go up by 5% a year for the 10 years of the project’s expected life.  Suppose that instead of maintaining the existing application, you could totally rewrite the application for $100,000 and that it would be ready in 1 year.  You would still have to contract for support for the old program for the first year for $12,000.  However, your expected support cost for the new application would be only $2,000 per year with an expected annual increase in cost of only 2.5% because the new system is written in a very common program language for which there is a large number of developers available for hire.  Which project would you choose?

One way to determine the answer to this problem is to calculate the workflows for some amount of time.  The following table shows a project with an expected 10 year life.

	Project 1	Project 2
Year	Annual Cost	Annual Cost
1	$     12,000.00	$   112,000.00
2	$       12,600.00	$          2,000.00
3	$     13,230.00	$        2,050.00
4	$       13,891.50	$          2,101.25
5	$     14,586.08	$        2,153.78
6	$       15,315.38	$          2,207.63
7	$     16,081.15	$        2,262.82
8	$       16,885.21	$          2,319.39
9	$     17,729.47	$        2,377.37
10	$       18,615.94	$          2,436.81

You could just add up the cost for the first 10 years to see which has the lowest overall cost.  That would be ok if your money had no other potential for gain.  However, that probably is not true.  At the very least, you invest in some money markets or government guaranteed bonds to earn at least 3%.  Therefore what you really need to do is to discount those future costs using the opportunity cost of money (what you could earn from some other investment).   This is called calculating the present value of each annual cost value.  Summing these up over the life of the project results in the net present value (NPV) of the cash flow stream.

Now if we had a project that was making money, we would probably look which project option would result in the highest NPV.  However, because we are dealing with costs displayed as positive numbers, we want the project with the smallest NPV cost.  (Of course, to be proper, we probably should display all costs as negative values.  In that case, we still would want the largest value or the closest to zero when both projects have a negative NPV.)

The following table shows the addition of a Present Value column for each project along with a sum of all the present values at the bottom of the table.

	Project 1		Project 2
Year	Annual Cost	Present Value	Annual Cost	Present Value
1	$     12,000.00	$          12,000.00	$   112,000.00	$       112,000.00
2	$       12,600.00	$            12,233.01	$          2,000.00	$              1,941.75
3	$     13,230.00	$          12,470.54	$        2,050.00	$            1,932.32
4	$       13,891.50	$            12,712.69	$          2,101.25	$              1,922.94
5	$     14,586.08	$          12,959.54	$        2,153.78	$            1,913.61
6	$       15,315.38	$            13,211.18	$          2,207.63	$              1,904.32
7	$     16,081.15	$          13,467.71	$        2,262.82	$            1,895.07
8	$       16,885.21	$            13,729.22	$          2,319.39	$              1,885.87
9	$     17,729.47	$          13,995.80	$        2,377.37	$            1,876.72
10	$       18,615.94	$            14,267.57	$          2,436.81	$              1,867.61
		$       131,047.26		$       129,140.21

In this case, it actually will cost us less over a ten year period if we spend $100,000 today to rewrite the application and thus reduce the annual support costs than to continue to support the old system at its higher annual cost.

While this was a simple exercise to show you the basic concepts of how to determine when or if it is time to throw out an existing application and begin a new one, the concepts apply no matter now complex the cost structure may be.  The key is to not get distracted by the money spent in the past on an application.  That cost is not recoverable no matter what you do.  In economic terms, that is called a Sunk Cost.  Therefore, keep your eyes on what is important, the future and let only the future costs and benefits guide your decision.

Why is this the Tuesday rant?  Too many times I have read stories of companies and management who become so caught up in the money spent in the past for a project, that they refuse to consider the possibility that it may be time to cut their losses and try something else.  As a programmer, deciding to start over is almost like admitting defeat which is definitely not the nature of most programmers.  Do you know any cases where starting over can help you or your organization?

C’ya next time.

It Is Not About The Metadata!

A while back I was at a meeting in which the topic of discussion was how hard it was to find any documents on the SharePoint portal.  The site collection in question was created four years ago and while the initial structure of the site made sense aligning itself along department and function boundaries, things have changed since then.  More importantly, many of the original site owners have moved on and new site owners were never selected.  That was problem #1.

Users of the site were encouraged to place documents on the site.  However, they treated the site much like they would a network share dumping everything on the site in a series of nested folders within folders within folders within folders…  Well, you get the idea.  The site nesting was not much better.  As a result, getting to any file was a complex series of branching that could easily confuse the person looking for a file.

Problem #3 is related to problem #1.  Obsolete files and information from a year, two years, and even three years ago still sits on the site and has not been updated.  I say that this problem is related to problem #1 because when a site does not have an active owner, no one looks at the content of the site with a critical eye as to what should be there and whether the information has been kept up to date.

None of these problems can be solved by magic.  If only that were true.  Switch to a new software product will never fix a problem if the users continue to try to work and do things like they always did before.  New systems give you an opportunity to change the way you work.  Hopefully this leads to better systems.  But trying to force new systems to work the same as old systems and then claim that the new system is no better than the old system is just plain silly.

So why not just use the SharePoint search functionality to find the files?  Great idea if you craft your search based on moderately unique words or phrases found within the document, something I do all of the time.  The problem is that it takes more creativity that perhaps an equivalent search using Google.  But that is not a fair comparison either because Google constantly performs analytics over millions of searches to determine which searches are successful and then they rank return results based on that information among other techniques.

Unfortunately, doing the same thing in SharePoint requires additional work that many smaller organizations do not have time or resources to do.  So the solution I offered was to restructure the sites and folders to flatten the structure considerable while at the same time adding metadata to libraries to classify the documents.  Yes, it would take time to determine what that metadata should be.  But once defined, we could add that information back as managed metadata to help classify future documents.  After all, search can also take advantage of metadata to help narrow your search.

But the answer I got was that metadata was ‘too hard’ for the system’s users to figure out and manage.  Then the person who said that it was too hard started to pitch a different system in which people could store the data that would make it easier to find individual documents.  After moving most of the documents out of SharePoint into their new system, they would consider restructuring what was left in SharePoint.

The whole argument of ‘too hard’ struck me as odd coming from a professional. I remember a time many years ago when I was told by a secretary that using a word processor was ‘too hard’ compared to using her trusty typewriter.  Fortunately, the department head at the time believed in the future of PCs and convinced the secretary to just try it.  Of course, the test was rigged a bit.  The department head asked the secretary to type a three page letter that needed to get out and then proceeded after each time the secretary finished to edit the resulting letter to add, delete or otherwise modify the letter.  Using a manual typewriter meant retyping the entire letter.  Next he took a progress report to be sent to management and asked the same secretary to use the PC to type the report and then print a copy for his review.  Again he mercilessly modified the report.  But this time, the changes could be made quickly by just entering the changes, not retyping the entire report.  After a ‘few more’ documents were created both ways, the secretary actually came to the manager asking to have the typewriter removed so she would have room on her desktop for the computer and keyboard.

Another example of ‘too hard’ is when we tell children that math and science is ‘too hard’.  If enough people tell children this lie including peers and adults, they begin to believe it and stop even trying.  In fact, they begin to use the ‘excuse’ that it is ‘too hard’ when they don’t do well on homework and tests.  Since many adults believe math is too hard, they accept the excuse and give the child a pass on their poor grades.

I recently was having a conversation with another professional about my same age and when I told him what I do for a living, his response was, “Isn’t that too hard for a person our age?”  Really?  If it were socially acceptable, I would have slapped some sense into him right then and there.

But saying that metadata is too hard is not what really ticked me off.  A few days after this initial discussion about what to do to better organize the files in this SharePoint site, the person who told me that metadata was ‘too hard’ pitched their concept to management using this other non-SharePoint tool that would use ‘metadata’ to classify the documents to make them easy to find.

Stunned! It is really not about the metadata is it?  It is really not about how hard or how easy it is for users to learn a new system is it?  It is really not about the best use of corporate resources and using what one has to the best of its abilities is it?  It is really just about pushing a personal agenda.

C’ya next time.